14

Are CSS and templates allowed in robots.txt?

Should it cause any problems?

In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put disallow in robots for CSS, templates, etc. for my upcoming websites.

Improve this question
1
  • I don't think Joomla blocks these files by default - if that is what you are implying?
    – MrWhite
    Commented Oct 31, 2014 at 13:16

3 Answers 3

Reset to default
22

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would.

If you block CSS or JS files, it could harm how well your website performs in the rankings.

More info here: Updating our technical Webmaster Guidelines and here: Webmaster Guidelines

As this is a recent recommendation, many websites and CMS's (such as Joomla) will often have such files blocked in robots.txt. The reasoning behind this was usually that search engines did not need to crawl or index these files, so to stop unnecessary files and directories from getting indexed and to save 'crawl budget', these would often be blocked in robots.txt.

Improve this answer
7
  • 2
    Google have actually been saying for quite a long time that you should not block JS and CSS (Matt Cutts video from March 2012) as it could harm G's ability to crawl your site, it's just that they have made it "more official" recently.
    – MrWhite
    Commented Oct 31, 2014 at 15:25
  • 2
    It has been advised not to block CSS and JS for well over a decade. I know that rendering by search engines seems new, but it is not and some forms of rendering including simple JS has existed for a very long time now.
    – closetnoc
    Commented Oct 31, 2014 at 16:36
  • 1
    Additional note (somewhat related since it's often used with JS) - In regards to any controllers making a route accessible via theme/view/JSON/URi - you should block those or you may face non themed areas being indexed instead of the page using said data facet. This is an area of "view" that is valid to block, especially things that JS uses like JSON. Often G will run the JS, see the JSON link, and visit it. Causes errors or if it themes out, a half built index. Thats a whole new can-o-worms, but just be aware.
    – dhaupin
    Commented Oct 31, 2014 at 20:54
  • 1
    I felt compelled to express my strong disagreement with the cargo cult attitude that whatever Google says, we must do. Google also says your site will rank higher if you allow them to crawl images, use HTTPS, etc. By complying with everything Google says, you are just handing them more power to make bigger demands. At the end of the day, are your building your website for humans or robots?
    – Question Overflow
    Commented Nov 1, 2014 at 3:51
  • 1
    Guess it depends on how much you rely on good rankings in Google? Although like most things, its only a small indicator ina large number of factors they have. Like you mentioned they recently said HTTPs is a small ranking factor, yet I haven't changed any of the 100's of sites I work on to HTTPs, and their rankings are fine.
    – Max
    Commented Nov 1, 2014 at 6:52
3

The strategy for your robots.txt should always be answered by the question: which sections of my web must not be crawled by a robot and which sections may be crawled by a robot.

Robots implement their own logic and have multiple purposes (not only Google has a crawler...) so if you're assuming that a robot get's somehow "distracted" by your CSS and JS files you open up the robot's black box and assume what the current implementation and the current intent of the robot is. This is no useful long term strategy.

Instead of thinking in the robot's domain try thinking in your web's content domain.

I want to point out that a robots.txt file is no security mechanism.

Improve this answer
1

Templates are used by the blogging or content management software (CMS) application and should not be accessible externally where as CSS is read by the browser and search engines and should be read. Having said that, I would not block either one but I also would not change any that are by the CMS. Search engines and bots do not care about your templates. Just block access to your site by URL/URI that you do not want indexed or read but do not think of robots.txt as a security tool. It is not designed for that.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.