2

When URLs no longer exist on my site, I give a page and issue an HTTP 410 status (the GONE error).

I created a script that allows my co-administrator to add and remove pictures from the site and when pictures are removed, that picture page will instead produce an error page and return an HTTP 410 status code to the browser and within approximately five seconds, the sitemaps are updated to reflect the changes.

I also have every page set with the robots noarchive metatag like so:

<meta name="GOOGLEBOT" content="NOARCHIVE">
<meta name="ROBOTS" content="NOARCHIVE">

Everyday when I visit webmaster tools, I get a random number of error URLs all pointing to the photo pages.

Even though google is correct at stating they return 410 status code, URLs still appear in webmaster tools even though the co-adminstrator removed the photo pages through the script and thus all links on the site to the affected URLs as well as all references to the affected URLs from the sitemaps are removed.

I also noticed that the more errors google sees on my site (regardless of whether the status codes are 404 or 410) the less I make in adsense.

I also have configured googlebot to make a max of 10 requests per second (slider all the way to the right).

What can I do to lower the odds of google seeing the newly generated error URLs as a result of removing bad pictures from the site so that I don't see 410 status codes in webmaster tools?

Improve this question
11
  • For a while, I thought this was another How do I hurry Google the hell up?? question. It sorta is, but not really.
    – closetnoc
    Mar 17, 2016 at 3:08
  • Its more like how do I make some no-longer valid URLs evade the online police (aka googlebot).
    – Mike -- No longer here
    Mar 17, 2016 at 3:13
  • We know the web is not real-time. It was designed with that in mind. It would be nice if there was more that sitemaps could do. In your case, a sitemap could also contain URLs that have been deleted. It could reduce the 404s. G are you listening?? It could be cheaper to download sitemaps that have changed than try and fetch a bunch of pages that are no longer there.
    – closetnoc
    Mar 17, 2016 at 3:17
  • Yeah G! I begin to think now I should tone down my 404/410 error pages since bots run into them (especially potential hackers)
    – Mike -- No longer here
    Mar 17, 2016 at 3:20
  • I do not see a problem with hackers seeing a 404. It is like a 100, binary 4... count it on your fingers. ;-)
    – closetnoc
    Mar 17, 2016 at 3:22

1 Answer 1

Reset to default
2

You can't force Google to crawl/rescan your sitemap instantly.

Once Google index any of webpage from your website, it start crawling again and again i.e. they crawl webpages from their own indexed database. So if you removed those pages from sitemap, even from your website which you linked to, then still that pages will able to crawl for Googlebot.

Normally Googlebot crawl 404 pages often, because they think, webmaster know this error is encounter in their webmaster dashboard, so may be they will fixed one day, and hence they crawl 404 pages often, So 410 error(permanently gone) is good to display in specific case.

I think Unavailable after meta tags, is right solution for your website.

Improve this answer
3
  • You can ping Google to automatically/instantly resubmit your sitemap when it is changed. However, as you say in your second paragraph, the sitemap has nothing to do with Google reporting errors for pages that were previously indexed.
    – DocRoot
    Mar 17, 2016 at 12:23
  • I just read from the website you showed me this: "After the removal, the page stops showing in Google search results but it is not removed from our system. ". Also applying the unavailable tags to the remainder of my pages might cause google to constantly index/deindex my site. Anyone know the email to the CEO's of google? I want to make a rant.
    – Mike -- No longer here
    Mar 17, 2016 at 18:40
  • DocRoot@Ping/Fetch And Render tool works only when webmaster used for right purpose. Google will not attention to all type of request. Mike@That tags apply to specific page, not to whole website. I have not used that tag in my life, so I can't say anything, If by placing that tag, Google will stop crawling that page, then I think it is good solution for you. If you can't contact Google, then hangout event will be good fit for you.
    – Goyllo
    Mar 18, 2016 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.