3

I have content that was deleted several years ago and from time to time Googlebot still attempts to access those pages, filling up my logs with lots of 404, making the 'real' problems harder to find and to read.

I have found Google is still crawling and indexing my old, dummy, test pages which now are 404 not found already, but that question is more about removing pages from the index. My pages are no longer indexed, but I'd like Google to stop attempting to open them.

I also believe this is not related to How to effectively close a page?. The page was closed years ago (maybe we did it wrong back then), but I would like to prevent Google from still crawling those old URL's (they have already been removed from the index years ago).

Would a 410 work? Indicating that the content will not come back? Or is this something we have no control over?

Improve this question
2
  • Possible duplicate of How to effectively close a page?
    – Zistoloen
    Apr 8, 2016 at 11:24
  • I am aware of a 410, but my question is whether that will eventually stop Google from hitting the URL? Or if it will only remove the page from the index but Google will remain to load it from time to time?
    – Neograph734
    Apr 8, 2016 at 11:28

2 Answers 2

Reset to default
3

Yes you are absolutely correct. You need to throw 410 to indicate that you have permanently disabled the page from your site.

404 page does not indicate that to crawler. Assume that there is some temporary issue on your site due to which 404 is shown, you don't want crawler to remove ur pages from index in that case.

Also make sure you remove these pages from sitemap, also from any other linked pages + disallow them from robots.txt.

You can find a good fix by seeing from where your pages which are throwing 404 are linked to. You can identify those resources and remove the page links from there in addition to throwing 410.

Improve this answer
7
  • Even with a 410 status, Googlebot will come back and check sometimes. I've seen Googlebot check 15 year old removed URLs.
    – Stephen Ostermiller ♦
    Apr 8, 2016 at 13:39
  • Thanks @StephenOstermiller for adding to abhinsit's answer and confirming what I was afraid of. Just to be sure, if I add the path to robots.txt Google will never pick up on the 410. So I suppose I shouldn't do that?
    – Neograph734
    Apr 8, 2016 at 13:44
  • Yes I think you are correct .. If removed from robots it will not know the page's status has changed .. You can wait till all are crawled as 410, then add that to robots.
    – abhinsit
    Apr 8, 2016 at 13:46
  • It is a forum we've removed long ago, and Google is still attempting to fetch all the threads we had there. I am pretty sure this is based on a list Google keeps somewhere.
    – Neograph734
    Apr 8, 2016 at 13:49
  • 1
    If you can put them in robots.txt Googlebot won't crawl them. Presumably they have been 404 long enough that Google isn't indexing any of them.
    – Stephen Ostermiller ♦
    Apr 8, 2016 at 14:01
1

Better yet: 301 to the correct page.

Edit: since this forum is gone, 301 to a page that explains that the forum has been permanently shut down. That's because external links still point to interior pages of that dead forum. That's not your fault, but on the other hand you sure enjoyed collecting all that free link juice. Surely this has happened to you: you click a link that seems perfect, only to land at some jerk's homepage and go "WTH, this stinks." Back up, look at the URLs and find out the jerk destroyed the content you needed. And in that moment, who are you angry at? Bingo.

As far as Google mistaking that for doorways, no worries - people shut down forums all the time, and that's certainly better than leaving them up to be stuffed full of spam by robots. I don't know if your platform allows this, but the HTTP protocol allows you to serve a 404 that is both a redirect and an actual web page: include a Location: in the HTTP header, and an HTML header/body with the usual redirects.

The crawling continues because external sites still link to the old page location; or Google is aware of that page being bookmarked by users. That is absolutely free link juice that you have earned! If it brings enough traffic to be worth the bother, and the content is any good, restore the forum in archive mode.

Even if you 410, that doesn't mean Google will disappear from the logs forever; as long as those external links and bookmarks live, the Free Traffic Fairy (er, Google) will check up periodically to see if the URLs have Lazarus'd.

As far as the access log pollution, grep -v ’/forum-URL-pattern-here/.*404' Though you may want to know about organic traffic to those pages; it is worth money.

Improve this answer
4
  • These pages have no replacement pages. We shut down the forum and that was is. Redirecting to the homepage feels a bit like tricking Googlebot, which I suppose it won't like.
    – Neograph734
    Apr 9, 2016 at 9:43
  • That's easy, redirect to a page that says "we shut down the forum." Google issue solved. These are coming from links external to your website - links made by other webmasters or posters on other websites. I don't know if you've ever clicked a link like that, but it's very annoying and confusing when the link takes you somewhere unexpected without any explanation to the surfer of why the link didn't work. Your web site is not there to make your logs easier to read. It is there to bring customers to your company.
    – Harper - Reinstate Monica
    Apr 9, 2016 at 17:29
  • What advantage (besides the dubious one of sponging up residual link juice) do you think serving a 301 redirect to a "forum closed" page would have over simply serving a 404 / 410 error page with the same content?
    – Ilmari Karonen
    Apr 11, 2016 at 11:03
  • That's fine too, if you're comfortable that enough browsers support displaying content on a 404/410. That wasn't always so, some browsers would either not render or pre-empt the content with their own content. Either way, the goal is customer engagement, which is the ultimate goal of link juice and traffic. It won't help link juice enough to bother... The only thing that will is leaving the forum up, perhaps in an archive mode.
    – Harper - Reinstate Monica
    Apr 11, 2016 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.