Tag Info

Hot answers tagged

20

Google have very recently updated their guidelines to officially state that you should not block access to CSS or JS files in robots.txt. This ensures that when Google crawls the site, it can render it exactly as a browser would. If you block CSS or JS files, it could harm how well your website performs in the rankings. More info here: Updating our ...


3

The strategy for your robots.txt should always be answered by the question: which sections of my web must not be crawled by a robot and which sections may be crawled by a robot. Robots implement their own logic and have multiple purposes (not only Google has a crawler...) so if you're assuming that a robot get's somehow "distracted" by your CSS and JS ...


1

Templates are used by the blogging or content management software (CMS) application and should not be accessible externally where as CSS is read by the browser and search engines and should be read. Having said that, I would not block either one but I also would not change any that are by the CMS. Search engines and bots do not care about your templates. ...



Only top voted, non community-wiki answers of a minimum length are eligible