6

We recently added a sitemap to our site but didn't exclude unpublished items (which, unless you're logged in as an admin, will 404). Google started trying to crawl them and as a result, we saw a huge increase in the number of 404s in the Google Webmaster tools.

Increase in 404s

We've since corrected the sitemap and now those items are not included, however the graph still indicates there are a high number of 404s. We can mark these URLs as "fixed" in the webmaster tools, but they aren't fixed - they're still 404ing, they just shouldn't be crawled and indexed.

Is it possible to tell Google not to crawl these? Will I need to add these 40,000 links to our robots file? And will these 404s be causing an issue for SEO?

Improve this question
2
  • 3
    I'm changing your question from "indexing" to "crawling". Google never indexes 404 pages. The graph indicates that it is crawling the URLs, but does not indicate that it has included any of them in the search index.
    – Stephen Ostermiller ♦
    Commented Feb 26, 2016 at 11:48
  • 2
    Never mark a 404 error as fixed if the page is not supposed to be there. You are telling Google that the page should be there and to try again. Google will try a page several times over a period if it gets a 404. When you set a correctly issued 404 error as being fixed, you are starting the process all over again.
    – closetnoc
    Commented Feb 26, 2016 at 17:21

2 Answers 2

Reset to default
5

Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article about Google's big memory for 404 pages.

You say that you already removed the URLs from the sitemap. That is good. If you hadn't already done so, that would be the first step.

You don't want to add 40,000 individual URLs to your robots.txt file. That could produce a file that is too big. The max robots.txt size is 500KB for Google. Other crawlers may not process even that much.

Having "404 Not Found" error pages on your site is not a problem. Google's John Mueller says:

404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html

Having that many 404 errors will make that report in Google Search Console much less usable. One way to remove them from it would be to return a more appropriate status. I would suggest: "401 Unauthorized". That indicates that there is content there, but that the user would be required to login to see it. If the user is logged in, but not an admin, then a "403 Forbidden" status would be appropriate.

Improve this answer
2
  • "Googlebot generally never stops" - although you'd perhaps think that Google would be quicker to look the other way for a URL that had only ever appeared "briefly" in an (advisory) sitemap and had never actually returned a valid response? "don't want to add 40,000 URLs" - although with a good amount of luck it could be a one liner: Disallow /admin-only/ !?
    – MrWhite
    Commented Feb 26, 2016 at 17:00
  • Google will clear out it's URL index, but it takes a while and it will not clear out anything where there is still a link or anything other than a 404 or 410 is issued. Google will forget URLs and stop crawling, I know this because Google has successfully forgotten over 1 million of my URLs, but it does seem like they do not forget when links or other references still exist. For example, Google has just finally forgotten about some URLs after a decade! This is because spam sites created links to these resources until within the last two years. Funny huh??
    – closetnoc
    Commented Feb 26, 2016 at 17:28
1

Block the links using robots.txt.

This will work for a large set of URLs which have some kind of common origin URL.

A directory and its contents by following the directory name with a forward slash:

Disallow: /sample-directory/

From Google Search Console Help.

Improve this answer
3
  • little bit too broad answer, but straight into head. I don't know if it's possible because we are talking about 40,000 links for robots.txt file. I never saw that robots.txt file.
    – Josip Ivic
    Commented Mar 3, 2016 at 10:00
  • So... in my opinion, this isn't a solution. Try to edit an answer and put a reason why he should do this, I'm interested in approach because we're talking about bunch of links. :)
    – Josip Ivic
    Commented Mar 3, 2016 at 10:01
  • Apologies I assumed that those 40,000 links were from a common origin or has a some kind of a unique directory pattern.
    – varun
    Commented Mar 3, 2016 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.