I tried googling, but did not find a good resource. What are some guidelines about deciding what to disallow in robots.txt?

closed as primarily opinion-based by Rob, John Conde Sep 27 '18 at 15:18

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • That's up to you. What don't you want search engines and bots to crawl? – John Conde Sep 27 '18 at 15:19

Robots.txt should be used to disallow any public information that would cause problems if robots hit it:

  • URLs that change the state on the server. For example a hit counter or other user stats logger.
  • Site search because Google doesn't want to crawl or index site search results. See Matt Cutts: Search results in search results
  • Public information that isn't of interest to search engines. For example content that has been syndicated on your site but is also available in other locations on the web.
  • Public files that could confuse bots. For example any web app that creates very large number of URLs. Perhaps combinations of parameters or a web based game with infinite and ever changing pages.
  • Low quality or auto generated content that could reflect badly on the rest of your site if search engines were to see it.
  • Public information that you want to hide from search engines, but where you don't mind that users can easily get to the content. For example some AJAX content or a redirect service.

You shouldn't use robots.txt to protect private information. Search engines may index URLs even without crawling them if they find links to them. For content that should remain private you should password protect it and let bots crawl the URL and get a 403 error. Admin URLs and logged in content should NOT be disallowed in robots.txt.

Not the answer you're looking for? Browse other questions tagged or ask your own question.