Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have noticed that since I have disallowed images using the robots.txt in my Joomla site, when sharing an article in Facebook, the image will not be displayed. Why is that? Is it indeed related?

My robots.txt file:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
share|improve this question
    
When "sharing", presumably you are sharing a link to that article? And where are the images not displayed... on Facebook or in your article? –  w3d Jun 20 '13 at 20:50
    
Yes, I am sharing the article link. The images are not showing up in the facebook post. It seems facebook doesn't find an image in the article. –  opk Jun 20 '13 at 21:28
1  
What does your robots.txt look like? Maybe Facebook is obeying you robots.txt commands? –  Max Jun 21 '13 at 2:29
    
I'm thinking that my robots.txt disallows the facebook crawler as well... (User-agent: * Disallow: /administrator/ Disallow: /cache/Disallow: /cli/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /logs/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ –  opk Jun 21 '13 at 10:45
add comment

1 Answer 1

up vote 1 down vote accepted

As moobot suggests, it would certainly make sense that the Facebook bot/crawler (it needs to crawl your page in order to build/show an image of the page) obeys robots.txt and your robots.txt currently blocks all bots from crawling your images.

If the FB bot didn't obey robots.txt then it would potentially be a bad-bot and is likely to get blocked by many sites.

You can try adding an exception to allow only FB to crawl your images:

# All user agents not matched elsewhere
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

# Facebook exception - same as above, except allow images
User-agent: facebookexternalhit
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
#Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Note I have comment out Disallow: /images/ from the facebookexternalhit group, so images are not blocked.

Any one robot/crawler will match only 1 group within the robots.txt file. It will match the most specific group. Only if it doesn't match a specific group will it match the generic User-agent: * group (the order does not matter).

FYI, I believe the full user-agent string for the FB crawler is (although it can be both http and https):

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
share|improve this answer
    
Thank you very much guys for helping! I will try it. –  opk Jun 21 '13 at 12:27
    
One more question, will the facebookexternalhit part override the first one when it comes to facebook robot, or the generic user agent " * " will override the second? Maybe something like "User-agent: *exclude facebookexternalhit" in the first part would be needed? Thank you. –  opk Jun 21 '13 at 16:07
    
Yes (to the first part). The FB robot will only look at the facebookexternalhit group, all other robots will look at generic group. You don't need to exclude the FB robot from the first (generic) part, since it will only look at the more specific group. I've updated my answer to clarify this. –  w3d Jun 21 '13 at 17:01
    
ok, many thanks. –  opk Jun 21 '13 at 17:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.