There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs for this content to be indexed.
I do it like this:
User-agent: * Disallow: /
Now I have a new subdomain 'facebook' which I'll be using to host apps developed for Facebook. My question is, should I use the code above to also block this subdomain?
The app itself shouldn't be accessed and used through the direct URL (but it still works I believe, didn't test it though), it should be used through the canvas URL, which is something like
apps.facebook.com/app_name. I don't mind search engines to index this URL, it's the right one to index. But I don't think it makes sense for them to index something like 'facebook.mydomain.com/apps/app_name'.
Should I block search engines with robots.txt for Facebook applications or should I ignore it? Any good reason for allowing search engines to crawl it?