I would like for User-agents to index my relative pages only without accessing any directory on my server.
As initial thought, i had this version in mind:
User-agent: * Disallow: */* Sitemap: http://www.mydomain.com/sitemap.xml
- Is it correct to block all directories like that -
- Would still search engines be able to see and index my sitemap if i disallowed all directories?
- What are the best practices for securing the robots.txt file?
Here is a good tutorial for robots.txt
#Add this if you want to stop Alexa from indexing your site. User-agent: ia_archiver Disallow: / #Add this to stop duggmirror User-agent: duggmirror Disallow: / #Add this to allow specific agents User-agent: Googlebot Disallow: #Add this to allow all agents while blocking specific directories User-agent: * Disallow: /cgi-bin/ Disallow: /*?*