Craigslist, one of the now many online want ad sites, has disabled RSS feeds for
viewing search results. I wouldn't say I am a heavy craigslist user in general
but it's my go-to used car shopping site and I had a few searches going to keep
an eye on the current value of certain vehicles and to keep a look out for a
couple rarer things I am looking for. I did all this via RSS.
Years ago, I had written a script to generate an OPML file that had the RSS
feeds with my search query for each location craigslist supported. The rarer
things might have one for sale in the whole country at any random time so just
going to the website and doing local searches sporadically wasn't going to cut
it. I have to be able to search continuously. With RSS gone, I needed a new
approach. Setting up a saved search on the site requires a login (how many
hundreds of websites do I have accounts on? No) and, I imagine, would require
setting up a search for each location in the country. I was really going to
need to automate again.
So the new script scrapes the website of each state to get the list of cities
that have a craigslist site, and does a search on each one. Then the result is
scrapped to pull the items out and the they get formatted into an RSS feed. I
run the script out of cron. It would be nice to dynamically generate the feed
when my RSS reader asks for it but it takes several minutes to search across all
locations and many time out periodically. Also if I set the feed up in feed
readers on multiple systems, I don't want to be hammering craigslist and looking
suspicious.
It's written in Perl and is hacky and gross but seems to do the trick. Have to
let it run for a bit and check if it's missing anything. It does have some
shortcomings as a general-use tool. Craigslist results are paginated so I only
see the first page of results. But as long as I am checking frequently enough,
or searching for something rare enough, I shouldn't miss anything. On the web
site, each result is also on it's own page that you have to click through to
see, so the RSS articles don't have any information from the item's description.
I just fill the article with the title, price, and add the URL to the specific
listing as a link so a feed reader can open it in a browser.
