Tagged Questions

0
votes
1answer
536 views

Restricting crawler activity to certain directories with robots.txt

I would like to use robots.txt to prevent indexing of some parts of my website. I want search engines to index only the / directory and not search inside my controllers. In my robots.txt, I have ...
3
votes
2answers
192 views

How can I allow search engines to index my invite only website in ruby on rails?

I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication: before_filter ...