We're hit with over 10,000 404s on Google Webmaster all of a sudden. We've been getting them a few now and then, but not even close to this magnitude. Upon examining the items in this list of errors, we find most of them to be URLs that we've never seen. i.e. we don't have them; we don't know where they came from; we don't list them in our sitemap; and we're not even sure what the "self-explanatory" numbers mean in the URL.
Please take a look; it's quite interesting. There are two varieties I've found.
A typical set on our list of 10,000+ (
404: http://www.mystore.com/fr/catalog/product/gallery/image/117366/id/3477/ Linked from: http://www.mystore.com/fr/catalog/product/gallery/image/117367/id/3477/ Linked from: http://www.mystore.com/fr/catalog/product/gallery/image/117365/id/3477/
Slight variation from above (
404: http://www.mystore.com/fr/catalog/product/gallery/image/296725/id/58116/ Linked from: http://www.mystore.com/fr/catalog/product/gallery/image/296724/id/58116/ Linked from: http://www.mystore.com/fr/catalog/product/gallery/id/58116/
A couple of things I noticed.
We have Spanish (
es) and French (
fr) language stores set up, denoted in the URL, beside our English one. All of these errors are of either with
URLs in this order of words do not exist. So, from
Set A, a proper URL would be
fr/catalog/product/gallery/id/3477/for the main image of product with
entity_id3477 without the "image" and its following number.
Looking at one of these links, one would think that numbers refer to an
catalog_product_entityfor "id" and an
entity_idof one of the EAV tables like
catalog_product_entity_varcharfor "image". However, looking at
Set B, it seems like there are actually not even what they seem to be. Our product IDs stop around 56,000. i.e. there is no product with ID 58116. Furthermore, these "image number" don't correspond to anything in particular. e.g. ID 296724 doesn't exist in the image tables and in the varchar table, it points to a meta title. UPDATE, I was just told that we once had products with IDs beyond 56K in the past before we upgraded our Magneto installation. It actually seems like these numbers are from our previous database. When we updated Magento, we started from a new database as well.
Now we think we may end up disallowing them in
robot.txt because we haven't figured out what the problem causing this is.. But I'm not sure if these URLs can be blocked at all because they're not "real" links.
I'd like to see if anybody has an idea what causes this sort of errors, either from experience or even if it's just a guess. I'd be happy to provide more information. Let me know. (I did read in someone's forum post somewhere that Google bots will try to guess a relevant link. Could this related to what's happening somehow?)
UPDATE: I have updated this post since when I first posted it. It seems that Google bots have a retained a list of our previous links and trying to crawl them..