Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want crawlers to have access to http://www.example.com but not http://www.example.com/

My robots.txt is as follows:

User-agent: *
Allow: /$
Disallow: /

My site is in Google search results, but I am not coming up in Bing, Yahoo, etc. I have had the same robots.txt since last year, and I initially requested inclusion ~1 year ago, having also resubmitted the URL to those latter search engines several times since as well.

Is my robots.txt blocking those other crawlers? And if so, why not Google as well?

share|improve this question

3 Answers 3

This does not seem possible, although it may depend on your particular webserver, but I strongly doubt this.

As far as I know, www.example.com and www.example.com/ are the same thing. PHP certainly treats then that way too (if you use parse_url() on both you get exactly the same results).

The first rule of your robots.txt allows the root of the site to be accessed (slash or no slash) and the second disallows everything since / technically begins all paths.

What can make things even more confusing is that some browsers remove the final slash and others add them in the location bar. They do that because the send the same request in both cases, basically asking for / on www.example.com.

share|improve this answer

Yes. Both those URLs represent the same resource. Actually, the correct one is WITH the slash (http://www.example.com/) since you access www.example.com and want the data from / (i.e. the root.)

What I would suggest you do, so search engines "don't make a mistake" is for you to provide the canonical version:

<link rel="canonical" href="http://www.example.com/product.php?item=swedish-fish" />

See the Google reference here:

http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

share|improve this answer

I tested this using Google Webmaster Tools and can see that it allows GoogleBot access to http://www.example.com and ALSO http://www.example.com/ - However, it denies access to everything else!

You should create accounts with both Google Webmaster Tools and Bing Webmaster Tools to test out your robots.txt file and check if there are any other problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.