Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress search pages (as well as admin pages).

However, I want the Google Images to be able to crawl anywhere, and I also want the AdSense bot to be able to crawl anywhere.

Have I done it right? I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values. Does the position of things in the file matter?

####################################################
# ALLOW MEDIA BOT TO CRAWL ANYWHERE
#####
User-agent: Mediapartners-Google
Allow: /

####################################################
# ALLOW IMAGE BOT TO CRAWL ANYWHERE
#####
User-agent: Googlebot-Image
Allow: /

####################################################
# ALLOW MAIN GOOGLEBOT SPECIFIC ACCESS
#####
User-agent: Googlebot
Disallow: /forums/calendar/
Disallow: /forums/gallery/
Disallow: /forums/members/
Disallow: /forums/user/
Disallow: /forums/index.php
Disallow: /forums/*&search_tags=
Disallow: /search/
Disallow: /*?s=
Disallow: /wp-admin/
Disallow: /wp-content/
share|improve this question

2 Answers 2

Does the position of things in the file matter?

No. The groups can be in any order.

I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values.

Any one crawler will only look at one group in the robots.txt file. The crawler will only look at the most specific group, determined by the length of the match against the user-agent record.

So, providing you have a Googlebot-Image group then the Googlebot-Image bot will only match this group. All other Googlebots (inc. News) will match the Googlebot group.

Your robots.txt file looks OK to me.

share|improve this answer

Thing looks fine but why do you disallow /forums/gallery/? Is that a media gallery containing images? If it is then avoid disallowing that directory as a whole.

share|improve this answer
    
We're disallowing it as the content is very thin, and there's hundreds if not thousands of similar pages where only the image is different. However we do allow the Google Image bot access to crawl everywhere. –  Buckers Sep 3 '13 at 15:09
    
Do you think what we're doing is "best practice"? Or should we allow the standard googlebot to crawl our galleries, with such thin content? –  Buckers Sep 5 '13 at 12:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.