Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
3 Answers
After Panda 4 update (May 2014), blocking css & js files will kill your seo.
There is a nice case study here that shows how doing that can hurt SEO :
https://yoast.com/google-panda-robots-css-js/
Nowadays Google wants to see your pages as same as users, so it is important to let him to see your client side scripts and style sheets.
-
This is the correct answer! View also Matt Cutts video: youtube.com/watch?v=B9BWbruCiDc Commented May 21, 2018 at 16:24
-
Although I suspect that since it's a link in the page that is being crawled, Google will read it anyway. Commented Sep 25, 2022 at 22:00
You don't state the reason to do that, so the general answer is no.
JavaScript files are not usually crawled, search engines know they are there, but they usually don't do anything with them because they don't have any content for users. So there is no need to do that from the Search Engines/Crawlers perspective.
From the perspective of normal users, it makes less sense because users don't use the robots.txt file and if they want to check your JavaScript files, they can access them directly after checking your code.
About the specific question, no, it won't hurt your SEO in any way because is basically a rule that nobody would pay attention to it.
Potentially may affect you badly if you make a mistake and block a folder that has content, but assuming you won't make that mistake, there is no benefit or drawback.
-
Would a search engine bot that supports/runs JavaScript still access (not index) the JS files?– unorCommented Mar 27, 2014 at 16:36
-
3You should not block JavaScript from being crawled because search engines (ie. Google) DO crawl (and execute to a certain extent) these files and if your content is dependent on the execution of this JavaScript, in any way, then it could hurt your SEO. Matt Cutts (Google) - March 2012 - Don't block Googlebot from crawling JavaScript and CSS @unor Yes.– MrWhiteCommented Mar 28, 2014 at 0:07
-
Yes, they can detect and execute, for instance, AJAX request for content and follow forms activated by JS.– PatomaSCommented Mar 28, 2014 at 0:58
-
1
-
1@PatomaS: "if you want to prevent crawlers to execute the JS files, then it´s better to block them by oter meanings, like .htaccess." - Good bots obey robots.txt. By disallowing crawling of JS files in robots.txt you will prevent bots from executing these files.– MrWhiteCommented Mar 31, 2014 at 10:18
It would certainly hurt your SEO for code search engines.
-
what's a code search engine that is functional as of today? I absolutely don't want my code to be searchable.– getbuckCommented Mar 27, 2014 at 17:10
-