Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Getting many 404 errors on my website for the following URLs:


I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google sites. Did I miss an option? If it's not there, is there a workaround using Webmaster tools?

share|improve this question
The reason for GWT reporting 404's is so you can fix them. Attempting to mask them by disallowing in robots.txt does not fix anything, the 404 is still there. – w3dk Sep 26 '12 at 16:00
@w3d Generally, yes. But from the path it looks like these errors might be within an admin area, for whatever reason. It's pretty common to block access in that case. – Su' Sep 26 '12 at 16:22
@Su': this is exactly the case. For some reason there are so many admin pages being reported as 404 like that, it's just this one is the most common pattern out of all. – Neolisk Sep 26 '12 at 20:10

There's no need to disallow crawling of 404 pages -- having URLs that return 404 is absolutely fine & normal. Having 404 errors on URLs that do not exist will not affect your site's overall crawling, indexing, or ranking (see Do 404's hurt my site?).

Also keep in mind that by disallowing crawling of URLs like this, it can result in them actually being indexed (since we can't be sure of what's behind the URL). On the other hand, if the URL returns a 404, and if we can crawl it to see that, then we won't index that URL.

share|improve this answer
Are you suggesting that having 404 on admin/internal pages all the time is normal, even though they were never supposed to be indexed in the first place? Even though they contain a different keyword set (compared to the site content), which if added multiple times, can ruin my ranking, because Google will assume my content is garbage? – Neolisk Oct 3 '12 at 13:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.