Should RSS feeds be disallowed in robots.txt?

Question

I have RSS feeds on my site. I've decided to follow Stack Exchange and disallow my RSS feeds in the robots.txt.

I don't want search engine to display the RSS feed page to people, that's not really a good page to see for new visitors.

Are there any advantage to allow search engine to crawl the RSS? Or is it a general good idea to disallow it?

Answer 1

There are many reasons not to block your feed, but only you can know if they are relevant for you. For example:

• There may be bots that especially look for feeds, e.g., feed search engines.
• There may be bots that use feeds to discover new content.
• There may be other cases where bots would like to access your feeds, now and in the future.
• Some web search engines might index feeds resp. feed URLs, so that they can give it as a result if users search for example.com feed, site:example.com inurl:feed, etc.
• Some user agents, e.g., feed readers, might follow rules in robots.txt.

I think most search engines will not be confused when they find a feed containing similar content to the front page of the website, as feeds are very common (almost every blog has them, news sites, forums, …). Make sure to link them with rel-alternate and give the corresponding MIME type in the type attribute:

From the HTML5 spec:

If the alternate keyword is used with the type attribute set to the value application/rss+xml or the value application/atom+xml
The keyword creates a hyperlink referencing a syndication feed (though not necessarily syndicating exactly the same content as the current page).

---

If your feeds contains the same content (i.e., the same number of posts and the same or less of the content) from a page of your site, you could use the canonical link type as HTTP header:

Link: <http://example.com/>; rel="canonical"

But it should not be necessary.