Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs for this content to be indexed.

I do it like this:

User-agent: *
Disallow: /

Now I have a new subdomain 'facebook' which I'll be using to host apps developed for Facebook. My question is, should I use the code above to also block this subdomain?

The app itself shouldn't be accessed and used through the direct URL (but it still works I believe, didn't test it though), it should be used through the canvas URL, which is something like apps.facebook.com/app_name. I don't mind search engines to index this URL, it's the right one to index. But I don't think it makes sense for them to index something like 'facebook.mydomain.com/apps/app_name'.

Should I block search engines with robots.txt for Facebook applications or should I ignore it? Any good reason for allowing search engines to crawl it?

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Block it. If there's no reason for them to be there by directly accessing that URL then you should do your best to prevent users from stumbling upon it. You never know if there is a vulnerability or some other issue related to direct access of that content so if there's no need for someone to access it that way why even leave the possibility open for problems related to this? If that content is designed for access through Facebook then make sure that's the only way to get there.

share|improve this answer
    
As long as a user of the application is logged in to Facebook, I think they can use the application through the Canvas URL or through the direct URL. I'm not sure I can block access to the direct URL and keep the app working. The Facebook page for the app runs it through an iframe, blocking the direct URL will probably block the app at all, no? –  Ricardo Amaral Nov 10 '10 at 16:33
    
Just block it through robots.txt. If you find other issues arise where people are accessing directly and causing problems then put stricter controls in place. –  John Conde Nov 10 '10 at 16:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.