Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like for User-agents to index my relative pages only without accessing any directory on my server.

As initial thought, i had this version in mind:

User-agent: *
Disallow: */*

Sitemap: http://www.mydomain.com/sitemap.xml

My Questions:

  1. Is it correct to block all directories like that - Disallow: */*?
  2. Would still search engines be able to see and index my sitemap if i disallowed all directories?
  3. What are the best practices for securing the robots.txt file?

For Reference:

Here is a good tutorial for robots.txt

#Add this if you want to stop Alexa from indexing your site.
User-agent: ia_archiver
Disallow: /

#Add this to stop duggmirror 
User-agent: duggmirror
Disallow: /

#Add this to allow specific agents 
User-agent: Googlebot

#Add this to allow all agents while blocking specific directories
User-agent:  *
Disallow: /cgi-bin/
Disallow: /*?*
share|improve this question

migrated from stackoverflow.com Aug 10 '12 at 15:27

This question came from our site for professional and enthusiast programmers.

Do you mean "How to make a secure robots.txt file"? – w3dk Aug 10 '12 at 15:37
up vote 4 down vote accepted
  1. That's going to block your entire website from being crawled.
  2. No
  3. There is no such thing as securing your robots.txt. If you don't want to keep visitors out of your directory root you need to prevent that using more secure means. Putting a blank index.html file will easily do the trick. If you're running Apache you can also do it easily using htaccess.
share|improve this answer
Thanks John! For reference, here is a tutorial as well that has more info: askapache.com/seo/wordpress-robotstxt-seo.html – CompilingCyborg Aug 10 '12 at 11:32
As an extra reference for the visitor of this post, here is a another good tutorial that addresses my question above: esecurityplanet.com/trends/article.php/3856871/… – CompilingCyborg Aug 10 '12 at 11:47

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.