11

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky.

Is this the case and is there a way to allow just the preview bot to access images using robots.txt?

EDIT: It looks like the previews are generated both by the normal Google Bot and by an on the fly bot Google Web Preview as mentioned (briefly) on the Webmaster Central Blog.

By using a site: search and my monitoring software I could see when the bot hit my site and when this happened the images showed up just fine in the preview. So my guess is that the normal crawler ignores the images per robots.txt, but the preview crawler gets the images anyway.

This implementation seems kind of crumby because my options seem to be:

  1. allow google bot to crawl my images (which I don't want to do)
  2. use the nosnippet tag which blocks the preview, but ALSO snippets (which I don't want to do)
  3. Let the wonky previews appear which may adversely affect click throughs
Improve this question
2
  • If it's just a matter of not having the images indexed, you could allow crawling but serve the images with an x-robots-tag HTTP header with "noindex".
    – John Mueller
    Commented Nov 16, 2010 at 13:03
  • @John Mueller This looks like it is the answer. Why not post it in the answer section?
    – user42
    Commented Nov 16, 2010 at 14:04

3 Answers 3

Reset to default
3

I think John Mueller had it right in the comments.

If it's just a matter of not having the images indexed, you could allow crawling but serve the images with an x-robots-tag HTTP header with "noindex"

I didn't know that you could allow Google to crawl content w/o indexing it. I put his technique in place and am just waiting to get crawled to see if it worked.

I'll accept this as the answer in a few days unless John wants to add his comments to the answer section so he can earn the rep.

Improve this answer
2
  • Sorry about the comment-answer :). One thing to keep in mind is that this process is not really that quick at the moment. Changing the indexing status of images is generally slower than web-search, and updating preview images can also take much longer than updating the normal web-search content (cached page, title, snippet). In practice, I imagine this is something that will take on the order of weeks for you to see any changes - so be patient :).
    – John Mueller
    Commented Nov 28, 2010 at 20:47
  • Patience is key. Several weeks out from the change, some images are now visible, while there are still missing. But this appears to have been the solution to my problem.
    – user42
    Commented Dec 10, 2010 at 18:44
2

As the most part of the preview is done by the Google bot crawler, blocking crawling of some part of your site will impact the preview...

Why don't you want to allow Google bot to crawl your images?

Improve this answer
7
  • 2
    We spend considerable time and money investing in product photography and we'd prefer to keep our images out of image search as people who use image search are typically searching for an image and are not looking to buy a product. If our images were in the index it would make it easier for competitors to steal them.
    – user42
    Commented Nov 16, 2010 at 14:03
  • 1
    You can try to watermark your pictures and/or steganography a copyright inside and/or filter display by referral...
    – Pascal Qyy
    Commented Nov 16, 2010 at 21:15
  • 1
    @G. Qyy Or I could pay an infinite number of monkeys to scan the web for copyrighted images.
    – user42
    Commented Nov 17, 2010 at 21:47
  • @Jim: Or you can keep preciously you pictures on papers, and never, never put them on the internet... ^^
    – Pascal Qyy
    Commented Nov 18, 2010 at 12:21
  • 1
    Good point -- nothing can completely protect your pictures but I'd really prefer not to have them in Google's index. I am not a fan of watermarking, and the most cost effective way to reduce our images elsewhere is to keep them out of the index.
    – user42
    Commented Nov 18, 2010 at 13:53
1

The following is a technical solution that may or may not be simply to apply to your site.

It is possible (even likely) that Google will come out with a way to do this with just a few hints in meta data or robots.txt, but until then....

Step 1.

Create a redirection service/servlet for front page images.

I.e. an URL like

/frontpageimages/[image name]

that does a server side redirect to

/images/[image name]

Step 2.

Have all image links on your front page (and only front page) rewritten to go through the redirection service from step 1 rather than linking directly to the image.

Step 3.

Make sure that robots.txt allows googlebot to crawl /frontpageimages/

This should ensure that Google can crawl any images it encounters on your front page while leaving any images on other pages alone.

While the redirection service could (in theory) be used to crawl all your images without technically violating your robots.txt, it is not something that well behaved robots (like googlebot) are going to do. And ill behaved robots aren't going to worry about robots.txt.

Improve this answer
2
  • Who only wants their front page showing a preview? I know I want every page to be show a good preview. If you do this for every page you essentially are negating the purpose of blocking the images in the first place.
    – John Conde ♦
    Commented Nov 16, 2010 at 15:12
  • @John You are right. In that case, you either want Google to index you or you don't.
    – Kris
    Commented Nov 16, 2010 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.