Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want to allow full access to all Google

I am specially confused about Allow: /* below Disallow:

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google
Disallow:
share|improve this question
add comment

1 Answer

The Allow directive is a non-standard extension to robots.txt. The original robots.txt standard only includes Disallow directives. Google's robots do understand Allow directives, so it would be OK to use them in sections targeted at a Google robot. The Allow directive should always be used after a Disallow directive to counteract the Disallow. It can be used to give finer grained allowance (maybe just a single page) of entire section that would otherwise be disallowed.

In your case, the Allow directive is saying the same thing as the Disallow directive, so it may be omitted.

You say you that you want to allow all Google robots to crawl all of your site. If that is the case, you would need a rule for the main Googlebot as well. I would suggest using this in your robots.txt

# Google Image
User-agent: Googlebot-Image
Disallow:

# Google AdSense
User-agent: Mediapartners-Google
Disallow:

# Googlebot
User-agent: Googlebot
Disallow:
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.