Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Using a redirect statement in my htaccess file, people who type the following into the address bar...

http://example.com/index.php

...are redirected to...

http://example.com/

I also have a noindex, nofollow meta tag on all my website's pages.

My question is, given that redirect behavior and meta data, will googlebot index my mainpage (i.e. index.php) if my robots.txt file is as follows...

User-agent: Googlebot
Allow: /index.php
Disallow: /

User-agent: *
Disallow: /

If not, how should I change my robots.txt so that only my mainpage will be indexed in google and nothing more?

Thanks in advance!

share|improve this question

3 Answers 3

If you redirect to "/" but only allow "/index.php" to be indexed then nothing will be indexed. To you "/" and "/index.php" are the same page. But to the search engines they are two separate URLs. By blocking "/", which is the only page users will apparently see, you are blocking your whole website.

If you want to have finer control over what is indexed use the meta tags mentioned above and/or use the x-robots-tag HTTP header to control indexing. Specifically use X-Robots-Tag: noindex on all of your pages except for the index.php. Then there is no need to use the robots.txt file. I read somewhere the HHTP header is more effective and the prefered way to block pages from being indexed but I am unable to find that article at the moment.

share|improve this answer

It's risky. To be sure that Google does index your homepage make this:

User-agent: *
Allow: /index.php
Disallow: /a
Disallow: /b
...
Disallow: /z
Disallow: /0
...
Disallow: /9

So your root "/" will not match disallow rules.

Also if you have AdSense don't forget to add

User-agent: Mediapartners-Google
Allow: /
share|improve this answer

John Conde already explained that your current solution won't work (+1), as Googlebot will only evaluate the rules after being redirected already and obey Disallow: /accordingly to block all URLs including your root/main page thereafter.

However, for Googlebot specifically you can facilitate its extensions to the official robots.txt protocol (see section Pattern matching within Block or remove pages using a robots.txt file), like so:

User-agent: Googlebot
Allow: /$
Disallow: /

The $character is matching the end of a URL, so the Allow: /$ directive will allow any URL that begins with your domain name, followed by nothing, i.e. allow your root/main page after being redirected from index.php already.

The Disallow: / directive will block all other URLs as usual thereafter.

I've just tested this successfully with the robots.txt test tool in Google webmaster tools (available there via Crawler access under Site configuration), so you could confirm this yourself immediately likewise - looking into this invaluable selection of tools/information related to SEO in general and Google in particular is highly recommended anyway!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.