2

I want to allow full access to all Google

I am specially confused about Allow: /* below Disallow:

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google
Disallow:
1

The Allow directive is a non-standard extension to robots.txt. The original robots.txt standard only includes Disallow directives. Google's robots do understand Allow directives, so it would be OK to use them in sections targeted at a Google robot. The Allow directive should only be used when it is more specific than Disallow directive. This will counteract the Disallow. It can be used to give finer grained allowance (maybe just a single page) of entire section that would otherwise be disallowed.

In your case, the Allow directive is saying the same thing as the Disallow directive, so it should be omitted.

You say you that you want to allow all Google robots to crawl all of your site. If that is the case, you would need a rule for the main Googlebot as well. I would suggest using this in your robots.txt

# Google Image
User-agent: Googlebot-Image
Disallow:

# Google AdSense
User-agent: Mediapartners-Google
Disallow:

# Googlebot
User-agent: Googlebot
Disallow:
2
  • 1
    "The Allow directive should always be used after a Disallow directive" - Actually, if anything, it should be the other way round. Allow before Disallow. Whilst for the main search engine bots this doesn't matter, since the match is based on the specificity/length of the URL path. Some (old) bots supposedly use a "first match rule", whereby the first path matched wins. Having the Allow directive first, accommodates these others bots. Reference: blogs.bing.com/webmaster/2009/08/20/… – MrWhite Oct 28 '16 at 15:15
  • 1
    I removed the advice about where it should go relative to the other rules altogether. It doesn't seem relevant for this question since 1. it is Google, 2. I'm suggesting removing the rule. The link you give is great for in depth exploration of the issue. – Stephen Ostermiller Oct 28 '16 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.