Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that included in the search by the bots. I also want to point to my sitemap which gets auto generated by a PHP page. So here is what I have, what else should I include, and if I need to fix anything with it?

Also, it's linking to xmlsitemap.php because that generates the sitemap when called.

My goal is to allow any search bot crawl the forums to grab meta data.

User-agent: *
Disallow: /admin/
Disallow: /ali/
Disallow: /benny/
Disallow: /cgi-bin/
Disallow: /ders/
Disallow: /empire/
Disallow: /komodo_117/
Disallow: /xanxan/
Disallow: /zeroordie/
Disallow: /tmp/

Sitemap: http://www.vivalanation.com/forums/xmlsitemap.php

Edit, I'm not sure how to handle all the user's folders under /public_html/ since the robots.txt will be going in /public_html.

share|improve this question

closed as off-topic by Stephen Ostermiller Oct 20 '13 at 12:32

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because it is unlikely to help anybody else. Examples include individual website reviews for SEO purposes and general website troubleshooting." – Stephen Ostermiller
If this question can be reworded to fit the rules in the help center, please edit the question.

We don't offer individual site reviews here. I would be happy to re-open this question if you could narrow it down to a specific question about something you think might not be working correctly. – Stephen Ostermiller Oct 20 '13 at 12:33

If you have an FTP area for some users why not place their ftp folders outside of your web root so they cannot be indexed? Robots.txt is not a safe way to secure those folders. What is going to stop someone from linking directly to those files? What about robots which do not follow robots.txt rules? All you're really saying to those robots is please don't look here. And anyone viewing the robots.txt file can see which folders you've tried to restrict access to.

You should re-configure your FTP server to move the FTP folders outside of the web root or public_html folder.

At the least use htpasswd on those folders so users have to login to view the content. That would prevent bots from indexing the content of the folders and also prevent anyone from linking directly to files.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.