Is it possible to hide a page from users, but still have search engines crawl the content?
I have RSS feeds which are really messy and don't fit my theme, but they have content that I would benefit from having crawled.
-
Sounds like you should be using an xml sitemap instead– John Conde ♦Jul 24, 2013 at 23:57
Add a comment
|
1 Answer
I am not entirely sure the logic is correct here. If you have pages that you think are messy and not of use to users, generally it is spammy or poor content, and various search engines would have no need crawling those pages. When indexed, it could either start sending users there, or dock your websites overall ranking due to things like this.
In addition, showing different content to users and search engines is generally considered a form of black-hat SEO, and most search engines will treat it as such.
