8

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html

It states:

For optimal rendering and indexing, our new guideline specifies that you should allow Googlebot access to the JavaScript, CSS, and image files that your pages use. This provides you optimal rendering and indexing for your site. Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

By default, Joomla’s robots.txt file comes with disallowing:

Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Please advise, shall we remove below items from robots.txt file based on Google’s announcement?

Disallow: /components/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/

Is this is what is recommended as per announcement for Joomla based sites?

  • Why not just do away with robots.txt since nobody (not even search engines now that Google is making demands on what you should not disallow) is going to follow it anyway? – Question Overflow Nov 1 '14 at 3:59
  • Related (not a duplicate): Robots.txt - CSS allow or disallow – unor Nov 1 '14 at 11:29
3

Honestly you are better off removing everything from your robots.txt. As far as I can see, all PHP files in Joomla contain the line

defined('_JEXEC') or die;

Which means if you load a PHP file directly in the browser all you get is a blank file, which search engines will ignore. (They shouldn't ever come across these anyway unless you linked them directly.)

The problem with leaving some of these directories blocked is that some components and modules keep their CSS/JS files inside those respective directories and not in the preferred media or images folders.

So there is no reason to block any Joomla files from Google.

  • Thanks. However - i do see - when a fetch a page through Webmaster - its fetching fine - despite having disallow to all those folder. Is removing disallow will do any good to the pages ? – Gag Nov 1 '14 at 13:41
  • 1
    @Gagan I'm not sure but the fetching tool in Webmaster Tools probably ignores robots.txt. – DisgruntledGoat Nov 3 '14 at 11:11
  • 1
    GWMT does both. When you fetch as google it will show you how Google sees your site and how a user sees your site. @DisgruntledGoat is right, there is no need to block anything. – Brent Friar Jul 29 '15 at 11:59
2

Apart from the overall use/lack thereof, of robots.txt in a well managed Joomla site, with "good" third party extensions — the only places that should contain CSS, JS or images are:

/images
/media
/templates

and of course their sub-directories.

So, you could just remove those from robots.txt.

2

In Joomla 3.3, these lines have been removed from the robots.txt file :

Disallow: /templates/
Disallow: /media/

More info here : http://www.energizethemes.com/blog/joomla/have-you-updated-the-joomla-robots-txt-file.html

1

If you see your pages without errors when fetching as Google in WMT, then you're probably fine. But, in future, you might upgrade some content on your website, which will demand some scripts/css from some of blocked folders. Therefore, I think you might be better with allowing search engines to crawl all these folders containing CSS/JavaScript.

1

The most recent versions of Joomla no longer block the /media/ and /templates/ folders:

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

Not all extensions stick to the guidelines of where to place CSS and JS files etc, so a good work around is to allow Google to access these files regardless of where they are found.

You can achieve this by inserting a few lines to the start of your robots.txt file like this:

#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

EDIT:

Thanks @w3dk and @Stephen Ostermiller for the feedback! You are quite right. It is better to do something like this:

User-agent: *
Allow: *.css
Allow: *.js
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

Unfortunately this does not seem to work as intended because the longer (more specific) rules override the shorter rules and the allow lines are ignored. It doesn't seem to make any difference whether the allow lines follow the disallow lines or vice versa.

The only way I can seem to get around this is by doing something like this which seems to work when I test it in Webmaster Tools:

User-agent: *
Allow: /************************************************************.css
Allow: /************************************************************.js
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

EDIT 2 - BEST SOLUTION:

OK, so I did a little more research and found the answer at https://stackoverflow.com/a/30362942/1983389

It appears the most correct and most supported solution across all web crawlers is something like the following (allowing access to *.css and *.js files in the /bin, /cache, /installation, /language, /logs, and /tmp folders and possibly some of the other folders makes little sense):

User-agent: *
Allow: /administrator/*.css
Allow: /administrator/*.js
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Allow: /cli/*.css
Allow: /cli/*.js
Disallow: /cli/
Allow: /components/*.css
Allow: /components/*.js
Disallow: /components/
Allow: /includes/*.css
Allow: /includes/*.js
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Allow: /layouts/*.css
Allow: /layouts/*.js
Disallow: /layouts/
Allow: /libraries/*.css
Allow: /libraries/*.js
Disallow: /libraries/
Disallow: /logs/
Allow: /modules/*.css
Allow: /modules/*.js
Disallow: /modules/
Allow: /plugins/*.css
Allow: /plugins/*.js
Disallow: /plugins/
Disallow: /tmp/
  • 2
    This will allow Googlebot to crawl everything, which is quite different from the original robots.txt file - is that the intention? (However, this is the same as simply including Disallow: under the User-agent: Googlebot group, which would be more readable.) – MrWhite Jan 2 '16 at 10:26
  • Yes, the intention is to allow Google access to all CSS and JS files on the website. – Neil Robertson Jan 2 '16 at 10:30
  • 2
    Not just CSS and JS files, but all files on the website. (?) – MrWhite Jan 2 '16 at 10:31
  • 1
    w3dk is correct. If you add a special section for Googlebot, you have to duplicate all the existing rules in that section. Your proposed robots.txt file would allow Googlebot to crawl /logs/ while preventing other bots from doing so. – Stephen Ostermiller Oct 25 '16 at 20:08

protected by Community Oct 25 '16 at 20:03

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.