I would like for User-agents to index my relative pages only without accessing any directory on my server.

As initial thought, i had this version in mind:

User-agent: *
Disallow: */*

Sitemap: http://www.mydomain.com/sitemap.xml

My Questions:

  1. Is it correct to block all directories like that - Disallow: */*?
  2. Would still search engines be able to see and index my sitemap if i disallowed all directories?
  3. What are the best practices for securing the robots.txt file?

For Reference:

Here is a good tutorial for robots.txt

#Add this if you want to stop Alexa from indexing your site.
User-agent: ia_archiver
Disallow: /

#Add this to stop duggmirror 
User-agent: duggmirror
Disallow: /

#Add this to allow specific agents 
User-agent: Googlebot

#Add this to allow all agents while blocking specific directories
User-agent:  *
Disallow: /cgi-bin/
Disallow: /*?*
  • Do you mean "How to make a secure robots.txt file"?
    – MrWhite
    Aug 10 '12 at 15:37
  1. That's going to block your entire website from being crawled.
  2. No
  3. There is no such thing as securing your robots.txt. If you don't want to keep visitors out of your directory root you need to prevent that using more secure means. Putting a blank index.html file will easily do the trick. If you're running Apache you can also do it easily using htaccess.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.