I want Google to index the index page, but nothing else. Will this achieve it?
This question came from our site for professional and enthusiast programmers.
I think you should specify on Index page as follows:-
For other page, mention as follows:-
The correct way to block all search engines is using a robots.txt file, a simple text file placed in the root of your webserver containing the following code:-
This works for all files on your webserver rather than a spesific page like the metatag mentioned in @SivaCharan's answer.
Since not all spiders abide by the robots.txt protocol, you might also want to block a spesific crawler from your site if it is showing up in your server logs after being blocked in robots.txt.
You can achieve this using the
Add the following to the .htaccess file:-
This only applies to a single domain rather than the entire webserver.
If you have access to the httpd.conf you can add block a spesfic crawler from the whole server by listing the User Agent header fields there.
Include your new directives in the following section of Apache's httpd.conf file: