up vote 1 down vote favorite
share [g+] share [fb]

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am using .htaccess to redirect to /cgi-bin/index.cgi. I am currently using this:

User-agent: * 
Allow: /cgi-bin/index.cgi
Allow: /contact.html  
Disallow: /

However, google webmaster tools is saying:

Googlebot is blocked from http://mydomain.com/

Is there a way of allowing indexing of the root while blocking all other files i.e., mydomain.com/*

link|improve this question
75% accept rate
feedback

migrated from stackoverflow.com Aug 27 '11 at 21:23

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 4 down vote accepted

As suggested by Pekka you may want to try to place the Allow directives after the Disallow directives.

But given the differences in interpretations between Google, Bing and others, you may want to use a robots meta tag instead. This will be safer and more granular.

In your disallowed pages:

<meta name="robots" content="noindex" />

In your allowed pages:

<meta name="robots" content="index" />

(to be placed in your <head> tag)

See http://googlewebmastercentral.blogspot.com/2007/03/using-robots-meta-tag.html

link|improve this answer
+1 good suggestion. – Pekka Aug 27 '11 at 21:25
What's the point of the index meta tag? Can't you just put nothing? – glumbo Sep 2 '11 at 7:43
Yes you could just put nothing – user11715 Sep 2 '11 at 9:33
feedback
up vote 2 down vote

Maybe try it the other way round, put the disallow before the allow.

If the Wikipedia article on robots.txt is correct, it should work:

While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that Allow patterns with equal or more characters in the directive path win over a matching Disallow pattern.[8] Bing uses the Allow or Disallow directive which is the most specific.[9]

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.