Is it possible to hide a page from users, but still have search engines crawl the content?
I have RSS feeds which are really messy and don't fit my theme, but they have content that I would benefit from having crawled.
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this communityIs it possible to hide a page from users, but still have search engines crawl the content?
I have RSS feeds which are really messy and don't fit my theme, but they have content that I would benefit from having crawled.
I am not entirely sure the logic is correct here. If you have pages that you think are messy and not of use to users, generally it is spammy or poor content, and various search engines would have no need crawling those pages. When indexed, it could either start sending users there, or dock your websites overall ranking due to things like this.
In addition, showing different content to users and search engines is generally considered a form of black-hat SEO, and most search engines will treat it as such.