Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Because it sure looks like it does. For my sites we disallow the images directory and the previews are all missing images which makes the site look wonky.

Is this the case and is there a way to allow just the preview bot to access images using robots.txt?

EDIT: It looks like the previews are generated both by the normal Google Bot and by an on the fly bot Google Web Preview as mentioned (briefly) on the Webmaster Central Blog.

By using a site: search and my monitoring software I could see when the bot hit my site and when this happened the images showed up just fine in the preview. So my guess is that the normal crawler ignores the images per robots.txt, but the preview crawler gets the images anyway.

This implementation seems kind of crumby because my options seem to be:

  1. allow google bot to crawl my images (which I don't want to do)
  2. use the nosnippet tag which blocks the preview, but ALSO snippets (which I don't want to do)
  3. Let the wonky previews appear which may adversely affect click throughs
share|improve this question
    
If it's just a matter of not having the images indexed, you could allow crawling but serve the images with an x-robots-tag HTTP header with "noindex". –  John Mueller Nov 16 '10 at 13:03
    
@John Mueller This looks like it is the answer. Why not post it in the answer section? –  Jim Nov 16 '10 at 14:04

3 Answers 3

up vote 3 down vote accepted

I think John Mueller had it right in the comments.

If it's just a matter of not having the images indexed, you could allow crawling but serve the images with an x-robots-tag HTTP header with "noindex"

I didn't know that you could allow Google to crawl content w/o indexing it. I put his technique in place and am just waiting to get crawled to see if it worked.

I'll accept this as the answer in a few days unless John wants to add his comments to the answer section so he can earn the rep.

share|improve this answer
    
Sorry about the comment-answer :). One thing to keep in mind is that this process is not really that quick at the moment. Changing the indexing status of images is generally slower than web-search, and updating preview images can also take much longer than updating the normal web-search content (cached page, title, snippet). In practice, I imagine this is something that will take on the order of weeks for you to see any changes - so be patient :). –  John Mueller Nov 28 '10 at 20:47
    
Patience is key. Several weeks out from the change, some images are now visible, while there are still missing. But this appears to have been the solution to my problem. –  Jim Dec 10 '10 at 18:44

As the most part of the preview is done by the Google bot crawler, blocking crawling of some part of your site will impact the preview...

Why don't you want to allow Google bot to crawl your images?

share|improve this answer
2  
We spend considerable time and money investing in product photography and we'd prefer to keep our images out of image search as people who use image search are typically searching for an image and are not looking to buy a product. If our images were in the index it would make it easier for competitors to steal them. –  Jim Nov 16 '10 at 14:03
1  
You can try to watermark your pictures and/or steganography a copyright inside and/or filter display by referral... –  Pascal Qyy Nov 16 '10 at 21:15
1  
@G. Qyy Or I could pay an infinite number of monkeys to scan the web for copyrighted images. –  Jim Nov 17 '10 at 21:47
    
@Jim: Or you can keep preciously you pictures on papers, and never, never put them on the internet... ^^ –  Pascal Qyy Nov 18 '10 at 12:21
1  
Good point -- nothing can completely protect your pictures but I'd really prefer not to have them in Google's index. I am not a fan of watermarking, and the most cost effective way to reduce our images elsewhere is to keep them out of the index. –  Jim Nov 18 '10 at 13:53

The following is a technical solution that may or may not be simply to apply to your site.

It is possible (even likely) that Google will come out with a way to do this with just a few hints in meta data or robots.txt, but until then....


Step 1.

Create a redirection service/servlet for front page images.

I.e. an URL like

/frontpageimages/[image name]

that does a server side redirect to

/images/[image name]

Step 2.

Have all image links on your front page (and only front page) rewritten to go through the redirection service from step 1 rather than linking directly to the image.

Step 3.

Make sure that robots.txt allows googlebot to crawl /frontpageimages/


This should ensure that Google can crawl any images it encounters on your front page while leaving any images on other pages alone.

While the redirection service could (in theory) be used to crawl all your images without technically violating your robots.txt, it is not something that well behaved robots (like googlebot) are going to do. And ill behaved robots aren't going to worry about robots.txt.

share|improve this answer
    
Who only wants their front page showing a preview? I know I want every page to be show a good preview. If you do this for every page you essentially are negating the purpose of blocking the images in the first place. –  John Conde Nov 16 '10 at 15:12
    
@John You are right. In that case, you either want Google to index you or you don't. –  Kris Nov 16 '10 at 15:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.