---------------------------------------- | |
Ideas for an aggregator | |
February 06th, 2018 | |
---------------------------------------- | |
I want to build a new gopher aggregator, specifically for phlogs, | |
and I want it to work in a very particular way. I'm not sure this | |
is going to be a general solve for everyone, but if I can do it | |
right it will work for my personal use. | |
Version 1 Plan: | |
- It should know that a phlog has been updated, not a whole | |
gopherhole. | |
- I should provide it an array of gophermaps to keep track of. | |
- The gophermap will be hashed each day--store the hashes in | |
a file for simplicity, or a sqlite db-- and if the hash | |
changed from the previous day, that link is added to the | |
daily list. | |
- If the link doesn't resolve, ignore. It might be temporary. | |
- It should organize phlog updates by date so I can go back if | |
I missed a couple entries (like Bongusta) | |
- It should auto-update daily | |
- Output should be simple: | |
- Master gophermap is a symlink to the current month. | |
- Each gophermap is a year-month for easy archival | |
- Viewing the directory will list the archive and current | |
- Linking to current will include a link to the archive | |
(directory view) | |
Version 2 Plan: | |
- The link added to the aggregated list should point to the new | |
entry, not the main gophermap. | |
- Potentially do a diff on the gophermap if the hash has | |
changed and try to identify new links? | |
- Grab any new links and drop them in verbatim, potentially | |
- Relative links might need to be rewritten | |
- Problems with changed links? | |
- Would this be instead of linking to the map, or in addition | |
(sub links under a heading?) | |
- Allow a way for people to submit their phlog to the list and | |
make it a public system instead of a private aggregator | |
My plan was to implement this in Perl as a way to learn Perl. | |
I may still go ahead with that, but it might be easier for me to | |
build with my current knowledge in python. Regardless, if anyone | |
reading this wants to take a stab at it, go right ahead. I'd | |
rather have the tool than be the one to write it. | |