2

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress search pages (as well as admin pages).

However, I want the Google Images to be able to crawl anywhere, and I also want the AdSense bot to be able to crawl anywhere.

Have I done it right? I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values. Does the position of things in the file matter?

####################################################
# ALLOW MEDIA BOT TO CRAWL ANYWHERE
#####
User-agent: Mediapartners-Google
Allow: /

####################################################
# ALLOW IMAGE BOT TO CRAWL ANYWHERE
#####
User-agent: Googlebot-Image
Allow: /

####################################################
# ALLOW MAIN GOOGLEBOT SPECIFIC ACCESS
#####
User-agent: Googlebot
Disallow: /forums/calendar/
Disallow: /forums/gallery/
Disallow: /forums/members/
Disallow: /forums/user/
Disallow: /forums/index.php
Disallow: /forums/*&search_tags=
Disallow: /search/
Disallow: /*?s=
Disallow: /wp-admin/
Disallow: /wp-content/
Improve this question

2 Answers 2

Reset to default
4

Does the position of things in the file matter?

No. The groups can be in any order.

I'm now a little confused after reading that the addition of User-agent: Googlebot may override certain other values.

Any one crawler will only look at one group in the robots.txt file. The crawler will only look at the most specific group, determined by the length of the match against the user-agent record.

So, providing you have a Googlebot-Image group then the Googlebot-Image bot will only match this group. All other Googlebots (inc. News) will match the Googlebot group.

Your robots.txt file looks OK to me.

Improve this answer
0

Thing looks fine but why do you disallow /forums/gallery/? Is that a media gallery containing images? If it is then avoid disallowing that directory as a whole.

Improve this answer
2
  • We're disallowing it as the content is very thin, and there's hundreds if not thousands of similar pages where only the image is different. However we do allow the Google Image bot access to crawl everywhere.
    – pixelkicks
    Commented Sep 3, 2013 at 15:09
  • Do you think what we're doing is "best practice"? Or should we allow the standard googlebot to crawl our galleries, with such thin content?
    – pixelkicks
    Commented Sep 5, 2013 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.