There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs for this content to be indexed.

I do it like this:

User-agent: *
Disallow: /

Now I have a new subdomain 'facebook' which I'll be using to host apps developed for Facebook. My question is, should I use the code above to also block this subdomain?

The app itself shouldn't be accessed and used through the direct URL (but it still works I believe, didn't test it though), it should be used through the canvas URL, which is something like apps.facebook.com/app_name. I don't mind search engines to index this URL, it's the right one to index. But I don't think it makes sense for them to index something like 'facebook.mydomain.com/apps/app_name'.

Should I block search engines with robots.txt for Facebook applications or should I ignore it? Any good reason for allowing search engines to crawl it?

1 Answer 1


Block it. If there's no reason for them to be there by directly accessing that URL then you should do your best to prevent users from stumbling upon it. You never know if there is a vulnerability or some other issue related to direct access of that content so if there's no need for someone to access it that way why even leave the possibility open for problems related to this? If that content is designed for access through Facebook then make sure that's the only way to get there.

  • As long as a user of the application is logged in to Facebook, I think they can use the application through the Canvas URL or through the direct URL. I'm not sure I can block access to the direct URL and keep the app working. The Facebook page for the app runs it through an iframe, blocking the direct URL will probably block the app at all, no?
    – rfgamaral
    Nov 10, 2010 at 16:33
  • Just block it through robots.txt. If you find other issues arise where people are accessing directly and causing problems then put stricter controls in place.
    – John Conde
    Nov 10, 2010 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.