Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
Disallow: /

I want Google to index the index page, but nothing else. Will this achieve it?

share|improve this question
add comment

migrated from stackoverflow.com Feb 18 '12 at 14:01

This question came from our site for professional and enthusiast programmers.

2 Answers

Robots.txt

The correct way to block all search engines is using a robots.txt file, a simple text file placed in the root of your webserver containing the following code:-

User-agent: *
Disallow: /

This works for all files on your webserver rather than a spesific page like the metatag mentioned in @SivaCharan's answer.

mod_rewrite and htaccess

Since not all spiders abide by the robots.txt protocol, you might also want to block a spesific crawler from your site if it is showing up in your server logs after being blocked in robots.txt.

You can achieve this using the .htaccess file and mod_rewrite Here's an example for blocking Baidu and Sogou spiders:

Add the following to the .htaccess file:-

RewriteEngine on 
Options +FollowSymlinks 
RewriteBase / 

RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC,OR” 
RewriteCond %{HTTP_USER_AGENT} ^Sogou
RewriteRule ^.*$ - [F”

This only applies to a single domain rather than the entire webserver.

httpd.conf

If you have access to the httpd.conf you can add block a spesfic crawler from the whole server by listing the User Agent header fields there.

Include your new directives in the following section of Apache's httpd.conf file:

# This should be changed to whatever you set DocumentRoot to.
#

... 
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bots
SetEnvIfNoCase User-Agent "^Sogou" bad_bots
SetEnvIf Remote_Addr "212\.100\.254\.105" bad_bot

Order allow,deny
Allow from all

Deny from env=bad_bots
share|improve this answer
add comment

I think you should specify on Index page as follows:-

<meta name="Robots" content="INDEX, FOLLOW" />

For other page, mention as follows:-

<meta name="Robots" content="NOINDEX, NOFOLLOW" />
share|improve this answer
    
what if, let's say, the pages are not in HTML? –  mauris Feb 18 '12 at 7:43
    
@thephpdeveloper: As far as I know, this is common concept should be followed for all the type of pages. Main settings should be done on Robot.txt –  Siva Charan Feb 18 '12 at 8:11
1  
@thephp all your pages are HTML. If you are talking about indexable files like images or PDFs then you'll need to add those to robots.txt. Put them in a subfolder and block those folders. –  DisgruntledGoat Feb 19 '12 at 20:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.