WEB-DL

I've become quite fond of my little ytb script that allows browsing
YouTube in ELinks or Less, made as a solution to the YouTube
website itself no longer displaying without Javascript. I kind-of
doubt that anyone else uses it, though it's there in the scripts
section, but there are lots of other text-only YouTube browsers
that weren't quite to my taste either.

The reason there are so many, and that I could find the time to
make my own, is all down to the youtube-dl project. It does all the
hard stuff of figuring out how to extract the information from
YouTube's API, and I think even more significantly - being
regularly updated to adapt to changes in YouTube itself. Then you
can just get it to spit out key info like the description, title,
and download length (either in plain text, or you can export to
JSON). So it's pretty trivial to take that raw data and wrap it up
in whatever interface you like.

But whenever I use it I just wish there were 100 other programs
like it for the other sites that require Javascript or full CSS
support. Shopping sites like Ebay, blogs based on common platforms
such as Wordpress, forums, news websites, weather, TV Guide. What
if they all had little tools that would just spit out the parts
that I want so I could build it into my own interface, so much more
usable than the barely-navigable mess of broken Javascript and CSS
that Dillo, Lynx, etc. would normally present? Little ebay-dl,
wordpress-dl, etc. programs that take a URL or (where appropriate)
search query and command-line options such as --get-description and
--get-title, as well as the option of dumping all the separate
parts into a machine-readable format.

There are some things like this besides youtube-dl. There's the
XMLTV project ( http://wiki.xmltv.org/index.php/Main_Page ),
various weather data grabbers for things like the wttr.in service,
and even the units_cur script that grabs the currency data for my
Gopher currency converter. The trouble is that they're very hard to
find, usually poorly documented (the format that units_cur spits
out isn't documented at all), and rarely share much in common so
far as the interface or output formatting. Some sites like the
Internet Archive have APIs that might be used, but one has to spend
the time figuring them out, and they could always change on a whim.

Youtube-dl is great in that it offers both plain-text output of the
requested data, as well as putting it all into a machine-readable
file. So lazy people like me don't even have to worry about parsing
the JSON output if they don't want to. One improvement would be to
cache the downloaded web page/s so that you can make independent
program calls for each bit of info instead of having to split the
plain-text output in a dodgy way based on content like I did with
ytb detecting the video-length line. It also attracts a lot of
developers (somehow, continuously revising code to adapt to sudden
changes to website structures doesn't appeal to me personally in
the slightest), though projects like XMLTV don't seem to be so
lucky.

So my proposal would be a common web-dl project that united a whole
lot of individual tools for downloading/extracting data from
websites. The tools would all try to work in a similar way and
provide similar sorts of output, and people would also be able to
share tools created on top of these "downloaders" such as
terminal-based site-specific browsers, and plug-ins for Dillo. When
a website changes, just the -dl tool needs to be adapted to keep
spitting out the same info as before to the program running above
it.

Granted it could all be avoided if commercial/trendy websites
weren't such a mess to begin with, and could actually be
conveniently navigated with lightweight and terminal-based web
browsers. But there's no way of turning back the tide on that at
this point unfortunately.

- The Free Thinker, 2021

2022-10-15:
I discovered the strangly-named Woob project, which seems to share
the same goal. Annoyingly the developers have chosen Python and Qt
as the common base, and I dislike both of those. It also seems to
cover remarkably few of the functions or associated websites that
I personally use, and as it's all Python I'm not interested in
contributing. Still, it's better than nothing, and I'll keep it
in mind for next time I want to view an image from Imgur or track
a parcel sent with DHL.

https://woob.tech/