21

I have set up my site with sample pages and data (lorem ipsum, etc..) and Google has crawled these pages. I deleted all these pages and actually added real content but in webmaster tools, i still get a lot of 404 errors Google trying to crawl these pages. I have set them to "mark as resolved" but some pages still come back as 404.

Furthermore, I have a lot of these sample pages still listed when i do a search of my site on Google. How to remove them. I think these irrelevant pages are hurting my rating.

I actually wanted to erase all these pages and start getting my site being being indexed as a new one but I read it's not possible? (I have submitted a sitemap and used "Fetch as Google.")

3 Answers 3

17

Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling Google to try again thus starting the process of elimination all over again.

Just let these pages 404 for a while and Google will stop looking for them and will drop the pages from the index. It will take time, but short of a 410 error, this is the easiest way. A 410 error would make the process faster, but it is harder to present a 410 error and a 404 is the default making it the easier and natural solution.

Your removed pages will disappear in about 30-60 days if you can wait. It depends on how often Google visits your pages. It can take longer, but once 404's are found, Google likes to first spot check the site, then depending on how many 404's there are, may spider your site more aggressively.

Using a sitemap actually does not generally fix any problems with the index. It only makes life simpler for search engines. It is never taken as the be-all end-all list of pages any site has. If a search engine reads a sitemap and still finds pages not listed in the sitemap, it will continue to index those pages.

One option if it makes sense to do, is to list these pages in your robots.txt file. If there aren't too many (meaning something you can do and your robots.txt file would not be too long), that would be a faster solution. Otherwise, I would just wait and let the 404 errors expire on their own.

One last word. You will be okay. Really. It will all work out very well for you if you are patient.

6
  • 1
    Adding 404 pages to robots.txt sounds like bad practice. It will only confuse the crawler and takes a lot of completely unnecessary housekeeping.
    – Dorus
    Jun 4, 2014 at 11:29
  • @Dorus Not at all. One has nothing to do with the other. Adding any page to the robots.txt file will remove the page from the index very quickly. As well, the search engine will not try and access the file and therefore no 404.
    – closetnoc
    Jun 4, 2014 at 14:12
  • 1
    As you say, if you add it to the robots.txt the search engine will not try to access to the page, but the page will still existing. So if some day you remove it from the robots the indexing will return. It is a better practice to let the 404 or the 410 do the work.
    – user44345
    May 28, 2015 at 11:16
  • @closetnoc What did you mean with it is harder to present a 410 error?
    – Evgeniy
    May 11, 2016 at 8:35
  • @Evgeniy A 404 error is what is given by default (Apache at least and older IIS). A 410 error would have to be intentional and requires some work to make happen. Technically, it is not a difficult task, however, it does require some expertise though not a lot. Cheers!!
    – closetnoc
    May 11, 2016 at 13:52
8

Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally.

To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it next. If you want it removed faster, return a "410 Gone" status instead. Google removes 410 pages immediately after crawling them instead of waiting a day. Google doesn't remove 404 pages immediately to prevent web masters from shooting themselves in the foot as described by Matt Cutts:

So with 404s, along with I think 401s and maybe 403s, if we see a page and we get a 404, we are gonna protect that page for 24 hours in the crawling system, so we sort of wait and we say maybe that was a transient 404, maybe it really wasn't intended to be a page not found.

Another method you could consider is redirection. 301 redirecting an old page to a replacement will prevent it from showing up as an error in Google Webmaster Tools. This is only possible if there is some new page for each of the old pages. Redirecting all the test pages to your home page won't help, because Google considers redirects to the home page to be "soft 404" errors that will still show up in that report.

Having 404 errors in Webmaster Tools won't hurt you. Having some 404 errors on your site may even help you because it shows Googlebot that your site is configured correctly. Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools:

HELP! MY SITE HAS 939 CRAWL ERRORS!!1

I see this kind of question several times a week; you’re not alone - many websites have crawl errors.

  1. 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
  2. In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
  3. What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
  4. You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it. http://support.google.com/webmasters/bin/answer.py?answer=2467403
  5. We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages. http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
  6. There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors. http://support.google.com/webmasters/bin/answer.py?answer=181708
  7. Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.
5
  • 1
    That has not been my experience at all. Google wants a fresh index and deletes pages fairly quickly. What I see that seems similar to what you describe, are where other sites using the Google API in the past, do not refresh their data and will cite your older work. These are often spammy/junk sites and these citations can appear/remain/disappear/reappear. The primary reason I see for this is because the Google API used to be rather promiscuous and is no longer thus spam data is much older because newer data is very hard to come by especially if you have spammed in the past.
    – closetnoc
    Jun 3, 2014 at 17:51
  • 1
    Googlebot has a crawl mode that I call "we found a box of URLs in the basement". In this crawl mode it may crawl a thousand URLs from your site in a row, none of which you have used in years. The URLs usually have no inbound links, even from scraper sites. They are crawled in order of length, shorter URLs are crawled first. Jun 3, 2014 at 17:58
  • That may be true. Google is clearly big data. Any large database has noise in it. That is unavoidable. That may be what you are experiencing. It is possible that various databases are being reconciled. That makes sense. But I also warn you that junk sites can appear for only 2 hours with old links and old citations. I see this daily. They are in Russia and Poland primarily. These sites are used to game local search engines, but effect traffic to any site and can be picked up by Google. I get about 12 of these in my database every day. Generally, only 1 out of 12 sites remain for any period.
    – closetnoc
    Jun 3, 2014 at 18:04
  • What's a 939 error?
    – gnicko
    Oct 23, 2018 at 17:50
  • 939 is the number of errors, it isn't a type of error. Oct 23, 2018 at 19:50
5

Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of a 404.

Alternatively you could serve a 410 Gone instead. This is a much stronger (ie. deliberate) signal that the page has literally "gone" and is not coming back. This could prompt Google to remove the page from the SERPs sooner.

I have set them to "mark as resolved" but some pages still come back as 404.

They are only "resolved" if you have put the page back. If you mark it as resolved and the page doesn't exist then the crawl error will simply recur. If the page doesn't exist then just leave it as it is.

Genuine 404's don't harm your search ranking. The 404 report in GWT is primarily for your benefit so you can see when things go wrong... when pages can't be found that should be found!

These irrelevant pages in the SERPs are perhaps a minor annoyance to your users, however, what are they searching for to find your lorem ipsum?

Not the answer you're looking for? Browse other questions tagged or ask your own question.