How to Use the New Google Web Search RSS Feeds

How to Use the New Google Web Search RSS Feeds

Google's been the lone hold out among major search engines on RSS but the company quietly enabled feeds for web search results this week. The offering is pretty limited and frustrating, you have to go through Google Alerts to get an obscure RSS URL, but we offer a tutorial and some strategic advice in this post.

Web search RSS is useful for being alerted whenever search results for your keywords or link have changed; subscribing to least a few searches will let you know when Google users are seeing something new in the first few pages of search results for your company name, for example.

How to Get the Feeds

All the other major search engines make it really easy to grab a feed for any web search, but Google is probably concerned about spammers finding bizarre and unscrupulous uses for their feeds. We're all inconvenienced as a result.

To get a feed for a Google search you have to go to the web page for Google Alerts and set up an alert for your search. You can enter most queries here. including site: queries. (site:http://readwriteweb.com semantic for example.) You should select "web" instead of the default "comprehensive" if you're just interested in tracking web search results.

"Feed" isn't an option in the initial drop down menu of delivery options, you've got to select email first. After you've done that, look at your collection of alerts and click to edit the one you want by RSS. At this point "feed" is an option in the drop down menu. Select it and you'll be shown an RSS URL. Throw that puppy in your favorite feed reader and you're ready to rock and roll.

The feed will deliver any new links that show up in the top 20 search results for your query. That's pretty limited, but most people don't look beyond the first 20 results anyway. That means that this is good for high-level reputation tracking but not very good for discovery of new, more obscure pages of interest.

The RSS URLs that Google gives you are based on an arbitrary number and don't contain the text characters of your query. That means you can't build more feeds by simply editing the URLs, you have to go back in through Alerts and repeat the proccess for every feed of interest.

More Advanced Options

Here's how we're using the new Google search feeds. We've grabbed feed URLs for searches for A. our names, B. our company name, C. our company URL and (just for fun) one for each of those three items without the other two. For example: "Richard MacManus" -readwriteweb -http://readwriteweb.com.

That gave us a small pile of feeds, which we then ran through our favorite RSS splicing and deduplication service (we used Yahoo Pipes but if you're not comfortable with Pipes then Feed.informer.com is really easy to use). We spliced all these feeds together, filtered for duplicates and then threw the resulting feed into our highest priority feed reading system.

Now we can track our high level reputations constantly, without being paranoid about it. We might do this for concept searches as well so that if someone new starts ranking really high for topics we specialize in (semantic web, RSS) then we'll know about them and never look ignorant at parties.

If we were interested in getting an RSS feed for Google web search for discovery, more than just reputation tracking, we might do an "advanced search," increase the results displayed from 10 to 100 and then use Dapper.net to scrape a feed of results from that page.

All of this is more complicated than it ought to be, but once you set up even the most basic feed options then you don't have to think about it again. Though it isn't perfect, we do appreciate Google making these feeds available.