Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
Are there any harms if my site send NOINDEX headers for all my static assets?
For image files, I refer to those valueless, e.g. background images, button images, etc.
Update: more background information
I have this concern is since recent Google said they also execute JS and they might fetch content via Ajax. So, for example, if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right?
(I work with Google's crawling & indexing teams) Let's start with the easy one: assuming you don't want these files used for indexing at all, you can just use the robots.txt file to disallow crawling of the scripts and responses. This prevents Googlebot from accessing the URLs, and would of course also prevent it from using them for anything else. The URLs themselves may be indexed (just as the URL based on factors like links to the URLs, without having been crawled), but they can't be processed or executed. Our robots.txt reference has more specifics for those who love details.
On the other hand, if you do want the content (and potentially the responses, etc) indexed: We recommend not using the noindex x-robots-tag HTTP header for embedded content (JavaScript, CSS, responses, etc.) if you'd like it potentially used for indexing. For example, if a JavaScript file uses an AJAX request to retrieve content from your server which is then displayed, all of those elements would first need to be crawlable (not disallowed by the robots.txt file), and we recommend that they aren't served with a "noindex" x-robots-tag HTTP header. We can't guarantee that we'll be able to process & index all JavaScript, but our systems are getting better and better over time.
In general, most sites don't link to their JavaScript or CSS files using normal HTML links, they just reference them within their HTML files. Because of that, for the most part we won't index those files separately anyway (and even if we did, your website most likely has more relevant content than what we find in the JavaScript files :)).
First things first. Google does not index certain file types such as CSS, JavaScript, images, and so on. But, you say, Google does index images. Well yes and no. It indexes images by using the text surrounding the image and the link values associated with the image. Yes, during growing pains while updating their capabilities, Google has indexed some of these files, but Google does not see it as content or directly valuable. This was discovered with many versions of the same JavaScript in it's index and the chatter began in earnest. I too found my CSS file indexed years ago, but it is long gone now. That is so 2009! It is old news that no longer applies. That is it. Nothing to worry about here.
Second, Google cannot access all Ajax elements- at least not Ajax enabled textbox(s) like what Google uses for search. In fact, they describe how to make Ajax elements crawlable here: https://developers.google.com/webmasters/ajax-crawling/. Google cannot and will not enter values into an Ajax enabled textbox searching for stuff. They actually say so. The scope of Google Ajax is well defined and began with Facebook and Disqus for comments. http://www.seroundtable.com/google-ajax-indexing-14241.html
Part of the confusion are early reports that Google executes JavaScript and can access Ajax. Where this is a true case is where JavaScript generates URLs. This was for three reasons; one to search deeper and not miss valid links, two to evaluate JavaScript for keyword loading, and three to check for JavaScript bloat. Later, other reasons were added such as page previews, and evaluating response times. It allows Google to index the full potential of a page. You can see the full scope here: https://www.distilled.net/blog/seo/google-stop-playing-the-jig-is-still-up-guest-post/
To be clear, so far Google is not abusing websites and you have nothing to worry about. It is not indexing elements that you would not want indexed. However, if you want Google and others, BTW- Bing does this too, from accessing these elements, then Google recommends putting in specific lines to each element/resource in your robots.txt file as normal. You can read the recommendation about half way down this page: http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html.
I would caution you against this however. Unless you are having a problem, I would not take any action. The reason answers the SEO part of your question. Blocking some elements may negatively impact Google and others from indexing your site properly and returning fully relevant results. If however, you have a JavaScript news element (real news and not site news), for example, that could in effect skew your search results. So in this case, you may want to restrict access to the JavaScript using robots.txt. This is where you have to put your thinking cap on. Take your time. There is no crisis to handle. When you have evaluated your site, then make informed decisions and take action. Mostly, you should not have to do anything at all unless a JavaScript result does not represent your content topic or Google and others are triggering a JavaScript and causing a problems.
TLDR; No
So, for example, if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right?
No, you seem confused what NOINDEX actually does.
NOINDEX: Allows crawling, following the links in it. Disallows indexing (which would automatically include NOARCHIVE and NOSNIPPET)
NOFOLLOW: Allows indexing(they usually don't but might). Disallows crawling, following links, showing snippet.
What Google recommends is to allow bots to "follow" the page. If you set a NOINDEX header tag, it allows Google to follow/crawl the page but not index it.
Therefore all you should do is NOT set "NOFOLLOW" tag. This would block bots to follow and parse the script. Setting NOINDEX tag is fine.
PS: It is equivalent to setting NOINDEX header tag to files such as "robots.txt" and "sitemap.xml". Because Google does index them quite frequently .
Example: www.google.com/search?q=filetype%3Atxt+%22robots%22&gws_rd=ssl
For more:
if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right?
You are right, this is not good. Google knows very well how to use your CSS and JavaScript files to represent your site better in search results.
See this lesson with Matt Cutts about CSS, JavaScript, crawling and indexing: https://www.youtube.com/watch?v=m293K8JR2j4
Nowadays, don't worry about it. Although we don't set the x-robot http header noindex for static file css, js, Googlebot know what file should be indexed or not. Except image, because the images can grab organic traffic for our site.
Btw, So many people still don't understand what is robots.txt file and noindex tag. Robots text file what should to crawl and what not to crawl, for indexing? back to our settings. While noindex tag is what should to index and what not to index.
Robots.txt file = Please come in my home, you can see this, this but not this, you dont allowed view this room.
Noindex tag (including x-robot http header): you may see the entire contents of this home but please don't open it to others (keep it secret) for this, this...
