3

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup:

<div class="header">
    <div class="navigation menu">
        <ul>
            <li><a href="index.php">Home page</a></li>
            <li><a href="products.php">Products</a></li>
            <li><a href="reference.php">Reference</a></li>
            <li><a href="about_us.php">About us</a></li>
            <li><a href="contact.php">Contact</a></li>
        </ul>
    </div>
</div>

The problem is, that google crawlers keep looking for the old pages, which I removed and aren't on the server.

I keep removing the crawl errors (marked as fixed) in the webmaster tools, but the crawlers keep attempting to crawl those pages.

I believe that the last website version is cached somewhere, so is there a way to remove it? (I don't have a lot of experience with google webmaster tools.)

Improve this question
3
  • 1
    Do you redirect the old URLs to their new equivalents?
    – Tim Fountain
    Commented Aug 17, 2013 at 12:25
  • @TimFountain I completely removed the old pages from the server, and I'm not using them anymore.
    – Vucko
    Commented Aug 17, 2013 at 12:30
  • This is just one more google garbage - notice that neither Bing, nor Yahoo do this indexing of 404 pages. You will under circumstances have to wait years until Google realised the page is gone and stops indexing it, it has nothing to do with links from other pages, it is just how bad google works. The only you can do to avoid it is never even go to the moment where you delete a page or post or change it without a 301 redirect immediately following the action. Google tries to culprit the users (us) with it but clearly it is a flaw in their algorythms or bing would do the same, and it does not
    – user118386
    Commented May 19, 2021 at 9:45

2 Answers 2

Reset to default
5

This is going to happen for a looong time. Other sites might be linking to the old URLs, which will prompt Google to crawl them and/or your site might just be having problems (in Googles eyes) and is giving you the benefit of the doubt that the pages might return. Either way, Google continues to crawl old pages for a long time. It would be far worse for Google to suddenly stop crawling your pages after getting a bunch of 404's.

If the pages genuinely do not exist and there is no alternative then it is correct to return a 404 (Not Found). Or you can return a 410 (Gone) instead for these pages that are never going to return - that is a far stronger/definite indication to Google that the pages aren't coming back.

Note that serving a 404 (or 410) for these pages, and having them reported as such in GWT is not necessarily a bad thing, and will not detrimentally affect your site in itself. This is a private report for your benefit.

However, what can be bad for SEO is if there are other sites linking to your old page that would have otherwise passed PageRank. By returning a 404, these sites will no longer be able to pass that PageRank. If you have alternative pages for those removed, then 301 redirect to the new location in order to preserve your ranking, help search engines re-index your content and ... to please your users. But if you have simply removed the old content and not replaced it then you need to be prepared to take the potential SEO hit.

Improve this answer
5
  • So your saying that is ok that webaster tools gives 404 for those sites that aren't anymore on the server ? Does that effect in SEOing ( when I type the keywords in google search, my new website is ranked bad) ?
    – Vucko
    Commented Aug 17, 2013 at 12:45
  • 404's are completely normal if the page doesn't exist no more, the only time you should be worried is that if you have some good links point to pageB but pageB doesn't exist so no more so you lose out on those good links. So you'd 301 old link to new link, but only necessary if you have a large volume of links, otherwise 404 is perfectly valid.
    – Simon Hayter
    Commented Aug 17, 2013 at 13:01
  • And will those errors effect in SEO ranking ? And is there a way to somehow remove in other pages link that are redirecting to pages on my site that doesn't exist ?
    – Vucko
    Commented Aug 18, 2013 at 15:37
  • As bybe states above, the detrimental effect in SEO is if their are other sites that are linking to your old page. In returning a 404 you are no longer benefiting from any PageRank that the other sites would otherwise be passing to you. If you have a suitable new page on your site then you should 301-redirect requests for the old page to the new. (You cannot do anything else to influence the other sites links, other than manually informing them!) A 301-redirect will also help the search engines to update their index.
    – MrWhite
    Commented Aug 18, 2013 at 16:04
  • However, if you have chosen to remove the old content and there is nothing related that replaces it then that is a decision you have made and should be prepared to take the SEO hit. A 410 is probably the correct response in this instance. The fact that Google is reporting a 404 in the GWT report is not itself a problem in terms of SEO - if it is expected - that is what I was referring to in my answer. I'll update my answer to clarify this.
    – MrWhite
    Commented Aug 18, 2013 at 16:12
0

Maybe not the best or correct way to do this but I had the same issue. What I finally did was redirect to the homepage or a 404. I'd have to check to see if I still have the redirect but I no longer get the crawl error.

Improve this answer
3
  • 2
    This simply masks the error in GWT and does not benefit you. It is better to see the 404 in GWT - if it is indeed a genuine 404 with no alternative content. Redirecting, what should be a 404, to the homepage just confuses search engines and users.
    – MrWhite
    Commented Aug 18, 2013 at 16:46
  • @w3d That's what I said. I redirected to a 404. I confused you by also saying 'homepage' because one of the redirects needed to go to the homepage cause that's what it was for. (It's complicated)
    – Rob
    Commented Aug 19, 2013 at 3:57
  • 1
    Sorry, but that also confused me, since you don't normally "redirect to a 404". Often when people say that, what they are actually doing is an external redirection to a what looks like a 404 page and inadvertently returning a 200 status - which could potentially hide the crawl error in GWT (or it results in a "soft-404"). Apologies if I took your statement too literally, but it seemed ambiguous to me so I thought it was worth pointing out.
    – MrWhite
    Commented Aug 19, 2013 at 7:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.