We have a Magento based store.

Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files.

And I get 404 errors for them in Webmaster tools / Search Console.

After Googling and looking at examples, I settled on this as a good solution. (http://inchoo.net/ecommerce/magento-robots-txt-best-practices-google-image-crawler/)

I added this to robots.txt:

User-agent: Googlebot-Image
User-agent: *
Disallow: /media/

The problem now is:

We get a whole lot of "Blocked resources" and "Mobile usability" problems instead.

Google says that it could result in suboptimal rankings: https://developers.google.com/webmasters/mobile-sites/mobile-seo/common-mistakes/blocked-resources?hl=en

So how does one handle this so Google is happy? What would you recommend?

  • Does GWT inform you which resources are blocked? If they are .css and .js files (although I guess not in the "media" folder) then exceptions could be put in place. – MrWhite Aug 28 '15 at 9:44
  • Yes, actually our js and css files are combined and cached dynamically as well. And those are also served from the media folder it seems. This seems this could be a general problem for anybody that uses some sort of cached resources that change every once in a while. – user3401799 Aug 28 '15 at 13:18
  • I'm not very familiar with Magento, but blocking search engines from indexing your images, just for the sake of keeping GWT clean, seems like a bad idea to me. Image's can be a big factor for SEO, and having them at a uniform URL is important. Are there any Magento modules/plugins that can help handle this for you? Does it actually change image URLs just by regenerating them? Or is it something like the image isn't generated until the product page is loaded? – nathangiesbrecht Aug 28 '15 at 14:58
  • nathangiesbrecht - I do allow Googlebot-Image to crawl all images. All my googling hasn't turned up any modules for this issue. Generally the advice is "block /media in robots". Yes, Magento changes urls and filenames after cache is emptied. Images are generated and cached on request. On my way home I came to think that maybe the template is linking to these images with an a href and therefore turning up as 404 when changed. They open in a lightbox so maybe not really a need to have a link. I'll take a look at this on monday - I'll post my solution after a few days if successful – user3401799 Aug 29 '15 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.