Getting many 404 errors on my website for the following URLs:


I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google sites. Did I miss an option? If it's not there, is there a workaround using Webmaster tools?

  • 1
    The reason for GWT reporting 404's is so you can fix them. Attempting to mask them by disallowing in robots.txt does not fix anything, the 404 is still there.
    – MrWhite
    Commented Sep 26, 2012 at 16:00
  • @w3d Generally, yes. But from the path it looks like these errors might be within an admin area, for whatever reason. It's pretty common to block access in that case.
    – Su'
    Commented Sep 26, 2012 at 16:22
  • @Su': this is exactly the case. For some reason there are so many admin pages being reported as 404 like that, it's just this one is the most common pattern out of all. Commented Sep 26, 2012 at 20:10

1 Answer 1


There's no need to disallow crawling of 404 pages -- having URLs that return 404 is absolutely fine & normal. Having 404 errors on URLs that do not exist will not affect your site's overall crawling, indexing, or ranking (see Do 404's hurt my site?).

Also keep in mind that by disallowing crawling of URLs like this, it can result in them actually being indexed (since we can't be sure of what's behind the URL). On the other hand, if the URL returns a 404, and if we can crawl it to see that, then we won't index that URL.

  • Are you suggesting that having 404 on admin/internal pages all the time is normal, even though they were never supposed to be indexed in the first place? Even though they contain a different keyword set (compared to the site content), which if added multiple times, can ruin my ranking, because Google will assume my content is garbage? Commented Oct 3, 2012 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.