2

Getting many 404 errors on my website for the following URLs:

/system/app/pages/admin/revisions*

I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google sites. Did I miss an option? If it's not there, is there a workaround using Webmaster tools?

Improve this question
3
  • 1
    The reason for GWT reporting 404's is so you can fix them. Attempting to mask them by disallowing in robots.txt does not fix anything, the 404 is still there. – MrWhite Sep 26 '12 at 16:00
  • @w3d Generally, yes. But from the path it looks like these errors might be within an admin area, for whatever reason. It's pretty common to block access in that case. – Su' Sep 26 '12 at 16:22
  • @Su': this is exactly the case. For some reason there are so many admin pages being reported as 404 like that, it's just this one is the most common pattern out of all. – Neolisk Sep 26 '12 at 20:10
2

There's no need to disallow crawling of 404 pages -- having URLs that return 404 is absolutely fine & normal. Having 404 errors on URLs that do not exist will not affect your site's overall crawling, indexing, or ranking (see Do 404's hurt my site?).

Also keep in mind that by disallowing crawling of URLs like this, it can result in them actually being indexed (since we can't be sure of what's behind the URL). On the other hand, if the URL returns a 404, and if we can crawl it to see that, then we won't index that URL.

Improve this answer
1
  • Are you suggesting that having 404 on admin/internal pages all the time is normal, even though they were never supposed to be indexed in the first place? Even though they contain a different keyword set (compared to the site content), which if added multiple times, can ruin my ranking, because Google will assume my content is garbage? – Neolisk Oct 3 '12 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.