Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We're hit with over 10,000 404s on Google Webmaster all of a sudden. We've been getting them a few now and then, but not even close to this magnitude. Upon examining the items in this list of errors, we find most of them to be URLs that we've never seen. i.e. we don't have them; we don't know where they came from; we don't list them in our sitemap; and we're not even sure what the "self-explanatory" numbers mean in the URL.

Please take a look; it's quite interesting. There are two varieties I've found.

A typical set on our list of 10,000+ (Set A):

404: http://www.mystore.com/fr/catalog/product/gallery/image/117366/id/3477/

Linked from: http://www.mystore.com/fr/catalog/product/gallery/image/117367/id/3477/

Linked from: http://www.mystore.com/fr/catalog/product/gallery/image/117365/id/3477/

Slight variation from above (Set B):

404: http://www.mystore.com/fr/catalog/product/gallery/image/296725/id/58116/

Linked from: http://www.mystore.com/fr/catalog/product/gallery/image/296724/id/58116/

Linked from: http://www.mystore.com/fr/catalog/product/gallery/id/58116/

A couple of things I noticed.

  1. We have Spanish (es) and French (fr) language stores set up, denoted in the URL, beside our English one. All of these errors are of either with es or fr.

  2. URLs in this order of words do not exist. So, from Set A, a proper URL would be fr/catalog/product/gallery/id/3477/ for the main image of product with entity_id 3477 without the "image" and its following number.

  3. Looking at one of these links, one would think that numbers refer to an entity_id of catalog_product_entity for "id" and an entity_id of one of the EAV tables like catalog_product_entity_media_gallery or catalog_product_entity_varchar for "image". However, looking at Set B, it seems like there are actually not even what they seem to be. Our product IDs stop around 56,000. i.e. there is no product with ID 58116. Furthermore, these "image number" don't correspond to anything in particular. e.g. ID 296724 doesn't exist in the image tables and in the varchar table, it points to a meta title. UPDATE, I was just told that we once had products with IDs beyond 56K in the past before we upgraded our Magneto installation. It actually seems like these numbers are from our previous database. When we updated Magento, we started from a new database as well.

Now we think we may end up disallowing them in robot.txt because we haven't figured out what the problem causing this is.. But I'm not sure if these URLs can be blocked at all because they're not "real" links.

I'd like to see if anybody has an idea what causes this sort of errors, either from experience or even if it's just a guess. I'd be happy to provide more information. Let me know. (I did read in someone's forum post somewhere that Google bots will try to guess a relevant link. Could this related to what's happening somehow?)

UPDATE: I have updated this post since when I first posted it. It seems that Google bots have a retained a list of our previous links and trying to crawl them..

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.