2

When URLs no longer exist on my site, I give a page and issue an HTTP 410 status (the GONE error).

I created a script that allows my co-administrator to add and remove pictures from the site and when pictures are removed, that picture page will instead produce an error page and return an HTTP 410 status code to the browser and within approximately five seconds, the sitemaps are updated to reflect the changes.

I also have every page set with the robots noarchive metatag like so:

<meta name="GOOGLEBOT" content="NOARCHIVE">
<meta name="ROBOTS" content="NOARCHIVE">

Everyday when I visit webmaster tools, I get a random number of error URLs all pointing to the photo pages.

Even though google is correct at stating they return 410 status code, URLs still appear in webmaster tools even though the co-adminstrator removed the photo pages through the script and thus all links on the site to the affected URLs as well as all references to the affected URLs from the sitemaps are removed.

I also noticed that the more errors google sees on my site (regardless of whether the status codes are 404 or 410) the less I make in adsense.

I also have configured googlebot to make a max of 10 requests per second (slider all the way to the right).

What can I do to lower the odds of google seeing the newly generated error URLs as a result of removing bad pictures from the site so that I don't see 410 status codes in webmaster tools?

  • For a while, I thought this was another How do I hurry Google the hell up?? question. It sorta is, but not really. – closetnoc Mar 17 '16 at 3:08
  • Its more like how do I make some no-longer valid URLs evade the online police (aka googlebot). – Mike Mar 17 '16 at 3:13
  • We know the web is not real-time. It was designed with that in mind. It would be nice if there was more that sitemaps could do. In your case, a sitemap could also contain URLs that have been deleted. It could reduce the 404s. G are you listening?? It could be cheaper to download sitemaps that have changed than try and fetch a bunch of pages that are no longer there. – closetnoc Mar 17 '16 at 3:17
  • Yeah G! I begin to think now I should tone down my 404/410 error pages since bots run into them (especially potential hackers) – Mike Mar 17 '16 at 3:20
  • I do not see a problem with hackers seeing a 404. It is like a 100, binary 4... count it on your fingers. ;-) – closetnoc Mar 17 '16 at 3:22
2

You can't force Google to crawl/rescan your sitemap instantly.

Once Google index any of webpage from your website, it start crawling again and again i.e. they crawl webpages from their own indexed database. So if you removed those pages from sitemap, even from your website which you linked to, then still that pages will able to crawl for Googlebot.

Normally Googlebot crawl 404 pages often, because they think, webmaster know this error is encounter in their webmaster dashboard, so may be they will fixed one day, and hence they crawl 404 pages often, So 410 error(permanently gone) is good to display in specific case.

I think Unavailable after meta tags, is right solution for your website.

  • You can ping Google to automatically/instantly resubmit your sitemap when it is changed. However, as you say in your second paragraph, the sitemap has nothing to do with Google reporting errors for pages that were previously indexed. – DocRoot Mar 17 '16 at 12:23
  • I just read from the website you showed me this: "After the removal, the page stops showing in Google search results but it is not removed from our system. ". Also applying the unavailable tags to the remainder of my pages might cause google to constantly index/deindex my site. Anyone know the email to the CEO's of google? I want to make a rant. – Mike Mar 17 '16 at 18:40
  • DocRoot@Ping/Fetch And Render tool works only when webmaster used for right purpose. Google will not attention to all type of request. Mike@That tags apply to specific page, not to whole website. I have not used that tag in my life, so I can't say anything, If by placing that tag, Google will stop crawling that page, then I think it is good solution for you. If you can't contact Google, then hangout event will be good fit for you. – Goyllo Mar 18 '16 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.