Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is it possible to hide a page from users, but still have search engines crawl the content?

I have RSS feeds which are really messy and don't fit my theme, but they have content that I would benefit from having crawled.

share|improve this question
Sounds like you should be using an xml sitemap instead – John Conde Jul 24 '13 at 23:57

I am not entirely sure the logic is correct here. If you have pages that you think are messy and not of use to users, generally it is spammy or poor content, and various search engines would have no need crawling those pages. When indexed, it could either start sending users there, or dock your websites overall ranking due to things like this.

In addition, showing different content to users and search engines is generally considered a form of black-hat SEO, and most search engines will treat it as such.

share|improve this answer
good points. thanks – Andrew Findlay Jul 24 '13 at 23:03

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.