=================
SFTP, VoIP, tea
=================

Yet another personal news digest: on the work-related SFTP and
globbing adventures, attempts to get voice conferences working over
HTTP, and tea.


SFTP and glob
=============

Had to implement processing of files from a remote server, available
over SFTP. Already had local file processing implemented, so thought
SSHFS would be a good fit, with a systemd mount unit. Under the load,
it worked for a day, and then the processes disappeared without
leaving any logs, even with the "reconnect" option. Not completely
unexpected, since this sort of thing happens, especially with services
that leave no logs. Then learned that SSHFS is not maintained lately.

I also had FTP implemented, with curl, which supports SFTP as well, so
decided to go that route instead. Did not have globbing implemented
for FTP though, simply used a glob library that operates on local
files (while the FTP tasks worked fine with directories, rather than
glob patterns), but now it is needed, since the files relevant for
those tasks are not neatly placed in dedicated
directories. Furthermore, having about a thousand source objects
(separate processing tasks), traversing all the files (requesting
contents of all the directories in which they may be) for each one is
slow, especially over SFTP.

So I implemented custom glob pattern matching, with matching over
multiple patterns at once (in a single file hierarchy traversal), and
using a provided directory listing function, so that it can be used
for both local and remote files: ``glob :: Monad m => [(String, a)] ->
(String -> m [String]) -> m [(String, a)]`` (the ``a`` type is for
tagging of the patterns, and then of the found results).

It is left to add a dedicated thread that will run it, and some
synchronization to queue and block the worker threads until there are
results for them. Not a fan of synchronization between threads in
general, but threads are used here without needing synchronization
usually, as lightweight processes (since those thousands of tasks,
with a process per task, could be quite a bit of overhead, especially
given that hundreds have to run simultaneously), I probably would have
avoided multithreading anyway, if it was not for Haskell libraries
including and aiming blocking functions, and use of multithreading: it
is hard and mostly unnecessary to fight that.

Feels a bit odd to keep adding functionality like that: I also have
crontab and systemd.timer-style schedule string parsing there, for
custom scheduling, in-place pangocairo bindings for text rendering,
some logging and IPC helpers. Trying to keep the codebase relatively
small, but having to implement those seemingly generic things
anyway. Actually thought of extending an existing glob library or
writing a new one, but then the simplified matching (just character
sets and "*" are implemented here) will not suffice, and it is easy to
write in-place anyway. Likewise with schedule strings: they may look
potentially awkward, but are quite simple.


VoIP over HTTP
==============

In a post two months ago I wondered about using HTTP for voice
conferences in web browsers, to avoid complexity, bugs, and connection
issues of WebRTC, and tried it out recently, implementing
<https://codeberg.org/defanor/bwchat> for that.

For the chat implementation, decided to use C
(``-std=c89 -Wall -Wextra -pedantic``) with an optional libfcgi
dependency, and trying to keep the features minimal, making it usable
with simpler web browsers and audio players, since most of the web
chat, file sharing, and voice call software I am finding otherwise is
the opposite of that. Only supporting Ogg with the Opus codec (which
can be handled for such stream relaying even without depending on
libogg and libopus, at least in a somewhat hacky and potentially
unreliable way: simply resending the first Ogg page, containing the
identification Opus packet, and detecting the first page in a stream
by looking at the corresponding Ogg page header bit). Initially
implemented it in a single multithreaded process with ISO C threads
(including corresponding mutexes, condition variables), then rewrote
it to use shared memory (shm_open, mmap) and process-shared mutexes
and condition variables, with pthreads, then had to debug some FastCGI
business (tried to use the program in CGI mode, with fcgi-wrap, which
apparently fails to detect client disconnects, with write functions
always succeeding) and possible deadlocking, and decided that if it
causes trouble (at least complicates debugging, makes it less certain
where the issue is) already, it is better to avoid threads and their
synchronization, simply using a dedicated server to keep track of the
state and organize multiple clients. So rewrote it for the third time,
adding a server available over Unix domain sockets with SEQPACKET and
a few basic commands, simply passing the C structures over it, not
bothering with fancy serialization.

In the JS part, implemented the AJAX-based textual chat functionality
to add on top of regular JS-free chat, and implemented audio upload:
using MediaRecorder to obtain Ogg pages, but had to upload those in
separate requests, since Firefox does not support a streaming
upload. If it did though, could have simply used Icecast to relay such
streams: Icecast's source clients are supposed to provide a single
stream over HTTP, with the PUT method, and chunked transfer
encoding. Could still add relaying to Icecast into the chat though,
but it probably will not be very useful.

The Ogg pages are relayed without any synchronization and buffering,
and I flush the stream after sending those, so that part is fast. Had
to disable nginx's FastCGI buffering (by setting the X-Accel-Buffering
header, though it can be done in nginx configuration, too). Yet the
delays, when listening to it from Firefox with the <audio> element (as
opposed to, say, ffplay), are rather high: I think it simply does not
target live conversations. Maybe the buffering can be avoided with
more JS somehow, but for now I paused the project, since I am rather
short on spare time even without that.

Web development, both client-side scripting and server-side code,
feels just like it did about twenty years ago: hacks upon hacks in an
attempt to get something working with unsuitable technologies. For
some reason I thought that those times are behind, with the new fancy
browsers and their JS APIs, but apparently it is usually merely hidden
under layers of bloat.

Though speaking of VoIP and hacks, ICE and the negotiation felt the
same way while implementing Jingle calls in rexmpp, and network
connectivity in general keeps being annoying and tricky. I keep
imagining how simple protocols could have been if there was a nice
network layer to build upon, with encryption, optional authentication,
integrity checks baked in. So that one could just start streaming Opus
packets (or even a PCM data stream) over UDP on a well-known port, the
receiver would pick it up as they would with a phone. Combined with
static (and perhaps mobile) IP addresses (maybe obtained with a VPN),
there will be caller identification. A VPN protocol is likely to
handle encryption as well. Though even a simple stream of UDP packets
without any negotiation, using ISP-assigned IP addresses, would be
similar to phones, particularly landline ones (assuming static IP
addresses and no VPNs or additional relaying).

But networking is rather messed up with NATs, some governments keep
organizing anti-competitive ISP oligopolies or monopolies, and
threatening to block protocols used for VPNs. In such a situation one
has to keep using and adding hacks, to make things work somehow. I
think the best that can be done when adding hacks in general is to
make them easy to remove, revealing a sensible system hiding behind
them, once the underlying issues that led to them are sorted out.


Tea
===

After trying buckwheat and Dianhong teas recently, I decided to try
more kinds of tea, and this month tried rooibos tea, which reminded me
mostly of tobacco (of tar you get from a pipe in particular), and Jin
Xuan Oolong tea (aka Nai Xiang, aka Milk Oolong; though this one was
flavored, but I enjoyed it). And a couple of flavored black teas. So
far I tried dozens of regional coffee varieties, not remembering most
of them well, but teas seem to be more varied, and I drink more tea,
since it is easier to brew once for a whole day. Going to keep trying
those.

Today is a buckwheat tea day though, and it is enjoyable.


Other news
==========

- I keep slowly working through the physics textbook mentioned in the
 previous post, solving all the odd-numbered problems (for which I
 can check the solutions). It is going to take a while, but probably
 still worthwhile to be thorough about it. Solving many exercises
 slowly also provides spaced repetition, which is supposed to be good
 for learning. Maybe will try to approach it similarly to piano
 practice and other exercises: just keep doing it regularly, without
 strict goals, but hopefully improving eventually. Learning more
 about mathematics-related Python libraries while at it: I keep using
 org-mode with LaTeX previews, embedded code blocks, and images, but
 more often lately producing those sketches with matplotlib, rather
 than Inkscape. Especially if actual plotting is involved, of course.

- Visited an ophthalmologist because of eye dryness, turned out the
 conjunctivitis was not over, and they prescribed me topical
 antibiotics. I hope it is over now, after applying those, though the
 eye dryness did not go away. The doctor was grumpy about me reading
 on it online, as if it was some folk medicine (CDC, WebMD, Wikipedia
 referencing a meta-study posted at NIH, and localized
 recommendations all say that viral conjunctivitis is not a big deal
 and does not normally require antibiotics, while the doctor says
 that bacteria join it in a couple of days, and antibiotics are
 needed almost always; secondary bacterial conjunctivitis is also
 mentioned here and there online, as uncommon), which was
 awkward. Anyway, I gave it a try, but not sure what to do about eye
 dryness now; maybe will visit another ophthalmologist. And will have
 to visit a dentist for a regular check soon. Not pleasant things to
 deal with, and reminders that human bodies fall apart over
 time. Which is a good and useful property in several ways (speaking
 of which, there were no surprises on the recent presidential
 election here), but still an upsetting one in others.

- Tried making a pasta sauce closer to chili or shakshouka: featuring
 onion, bell peppers, and paprika, in addition to the usual garlic,
 olive oil, tomatoes, salt. It turned out nice, though took a while
 to cook (to caramelize the onions and ensure that the pepper skins
 are softened). Now planning to make more sauce like that, with
 eggplants added as well, but separately from pasta, and refrigerate
 it. To reheat it for pasta and other dishes later, multiple times,
 making up for the time it takes to cook.


----

:Date: 2024-04-24

You are viewing proxied material from thunix.net. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.