0

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that included in the search by the bots. I also want to point to my sitemap which gets auto generated by a PHP page. So here is what I have, what else should I include, and if I need to fix anything with it?

Also, it's linking to xmlsitemap.php because that generates the sitemap when called.

My goal is to allow any search bot crawl the forums to grab meta data.

User-agent: *
Disallow: /admin/
Disallow: /ali/
Disallow: /benny/
Disallow: /cgi-bin/
Disallow: /ders/
Disallow: /empire/
Disallow: /komodo_117/
Disallow: /xanxan/
Disallow: /zeroordie/
Disallow: /tmp/

Sitemap: http://www.vivalanation.com/forums/xmlsitemap.php

Edit, I'm not sure how to handle all the user's folders under /public_html/ since the robots.txt will be going in /public_html.

1
  • We don't offer individual site reviews here. I would be happy to re-open this question if you could narrow it down to a specific question about something you think might not be working correctly. Oct 20, 2013 at 12:33

1 Answer 1

1

If you have an FTP area for some users why not place their ftp folders outside of your web root so they cannot be indexed? Robots.txt is not a safe way to secure those folders. What is going to stop someone from linking directly to those files? What about robots which do not follow robots.txt rules? All you're really saying to those robots is please don't look here. And anyone viewing the robots.txt file can see which folders you've tried to restrict access to.

You should re-configure your FTP server to move the FTP folders outside of the web root or public_html folder.

At the least use htpasswd on those folders so users have to login to view the content. That would prevent bots from indexing the content of the folders and also prevent anyone from linking directly to files.

Not the answer you're looking for? Browse other questions tagged or ask your own question.