10

Are there any harms if my site send NOINDEX headers for all my static assets?

For image files, I refer to those valueless, e.g. background images, button images, etc.

Update: more background information

I have this concern is since recent Google said they also execute JS and they might fetch content via Ajax. So, for example, if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right?

2
  • 1
    Pointless because Google doesn't or shouldn't index JS/CSS assets. Commented Jun 8, 2014 at 9:08
  • 1
    @bybe Google do and should index JS/CSS assets. Commented Jun 11, 2014 at 5:56

6 Answers 6

9
+25

(I work with Google's crawling & indexing teams) - deleted most of the old answer :).

Three things:

  1. In general, JS & CSS files don't appear in normal search results, rarely even if you explicitly look for them with a site:-query. So probably this is a non-issue nowadays.

  2. If you do find non-HTML content from your site indexed (and don't want it), and it's ranking for normal queries that users make instead of your normal HTML pages, then you should fix the HTML pages first. If a CSS file is seen as being more relevant than your actual content, then the problem is not the CSS file.

  3. If after all of this, you still want to get those non-HTML files out of the search results, use the x-robots-tag HTTP header with noindex. The reference docs have all the details. This is a server-side header that you have to set, sometimes you can do it on a CDN too.

Let's start with the easy one: assuming you don't want these files used for indexing at all, you can just use the robots.txt file to disallow crawling of the scripts and responses. This prevents Googlebot from accessing the URLs, and would of course also prevent it from using them for anything else. The URLs themselves may be indexed (just as the URL based on factors like links to the URLs, without having been crawled), but they can't be processed or executed. Our robots.txt reference has more specifics for those who love details.

On the other hand, if you do want the content (and potentially the responses, etc) indexed: We recommend not using the noindex x-robots-tag HTTP header for embedded content (JavaScript, CSS, responses, etc.) if you'd like it potentially used for indexing. For example, if a JavaScript file uses an AJAX request to retrieve content from your server which is then displayed, all of those elements would first need to be crawlable (not disallowed by the robots.txt file), and we recommend that they aren't served with a "noindex" x-robots-tag HTTP header. We can't guarantee that we'll be able to process & index all JavaScript, but our systems are getting better and better over time.

In general, most sites don't link to their JavaScript or CSS files using normal HTML links, they just reference them within their HTML files. Because of that, for the most part we won't index those files separately anyway (and even if we did, your website most likely has more relevant content than what we find in the JavaScript files :)).

8
  • 1
    John, you may want to update this answer. "Blocked resources", unless I'm mistaken I wouldn't want to block stylesheets required for properly rendering my site.
    – Oskar Skog
    Commented Jul 19, 2017 at 9:20
  • I'm not sure what happened here, but the exact opposite is true... using robots.txt actively prevents Google from crawling those files, while noindexing allows Google bots to crawl them but NOT index them into SERPs. Commented Aug 17, 2022 at 19:26
  • @OskarSkog This is why John states: "This prevents Googlebot from accessing the URLs, and would of course also prevent it from using them for anything else." Commented Aug 17, 2022 at 21:31
  • 1
    Maybe it's just a poorly written answer, but it comes across like he's saying robots.txt is any easier way to prevent Google from indexing certain resources, but this is not accurate, the noindex header is easier and less risky (Google has said so in recent years). Likewise, even in 2014 when he wrote this answer, using the noindex header on e.g. CSS files did not prevent Google from indexing the parent HTML page. Commented Aug 17, 2022 at 21:51
  • 2
    What a blast from the past - so much has changed around JS on websites. x-robots-tag is definintely the better approach nowadays, but even that's generally unecessary now. I haven't seen JS or CSS files show up in normal search results in years. Commented Aug 19, 2022 at 11:12
7

First things first. Google does not index certain file types such as CSS, JavaScript, images, and so on. But, you say, Google does index images. Well yes and no. It indexes images by using the text surrounding the image and the link values associated with the image. Yes, during growing pains while updating their capabilities, Google has indexed some of these files, but Google does not see it as content or directly valuable. This was discovered with many versions of the same JavaScript in it's index and the chatter began in earnest. I too found my CSS file indexed years ago, but it is long gone now. That is so 2009! It is old news that no longer applies. That is it. Nothing to worry about here.

Second, Google cannot access all Ajax elements- at least not Ajax enabled textbox(s) like what Google uses for search. In fact, they describe how to make Ajax elements crawlable here: https://developers.google.com/webmasters/ajax-crawling/. Google cannot and will not enter values into an Ajax enabled textbox searching for stuff. They actually say so. The scope of Google Ajax is well defined and began with Facebook and Disqus for comments. http://www.seroundtable.com/google-ajax-indexing-14241.html

Part of the confusion are early reports that Google executes JavaScript and can access Ajax. Where this is a true case is where JavaScript generates URLs. This was for three reasons; one to search deeper and not miss valid links, two to evaluate JavaScript for keyword loading, and three to check for JavaScript bloat. Later, other reasons were added such as page previews, and evaluating response times. It allows Google to index the full potential of a page. You can see the full scope here: https://www.distilled.net/blog/seo/google-stop-playing-the-jig-is-still-up-guest-post/

To be clear, so far Google is not abusing websites and you have nothing to worry about. It is not indexing elements that you would not want indexed. However, if you want Google and others, BTW- Bing does this too, from accessing these elements, then Google recommends putting in specific lines to each element/resource in your robots.txt file as normal. You can read the recommendation about half way down this page: http://googlewebmastercentral.blogspot.com/2011/11/get-post-and-safely-surfacing-more-of.html.

I would caution you against this however. Unless you are having a problem, I would not take any action. The reason answers the SEO part of your question. Blocking some elements may negatively impact Google and others from indexing your site properly and returning fully relevant results. If however, you have a JavaScript news element (real news and not site news), for example, that could in effect skew your search results. So in this case, you may want to restrict access to the JavaScript using robots.txt. This is where you have to put your thinking cap on. Take your time. There is no crisis to handle. When you have evaluated your site, then make informed decisions and take action. Mostly, you should not have to do anything at all unless a JavaScript result does not represent your content topic or Google and others are triggering a JavaScript and causing a problems.

1
  • This answer is more intuitive than John's, and was rather insightful for 2014... many users see "Google employee" and upvote a poorly written answer. Commented Aug 18, 2022 at 7:33
5

TLDR; No

So, for example, if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right?

No, you seem confused what NOINDEX actually does.

NOINDEX: Allows crawling, following the links in it. Disallows indexing (which would automatically include NOARCHIVE and NOSNIPPET)

NOFOLLOW: Allows indexing(they usually don't but might). Disallows crawling, following links, showing snippet.

What Google recommends is to allow bots to "follow" the page. If you set a NOINDEX header tag, it allows Google to follow/crawl the page but not index it.

Therefore all you should do is NOT set "NOFOLLOW" tag. This would block bots to follow and parse the script. Setting NOINDEX tag is fine.

PS: It is equivalent to setting NOINDEX header tag to files such as "robots.txt" and "sitemap.xml". Because Google does index them quite frequently .

Example: www.google.com/search?q=filetype%3Atxt+%22robots%22&gws_rd=ssl

For more:

1
  • This answer is partly inaccurate... nofollow disallows following links only, it does not prevent crawling or showing snippets, at least not these days. To disable snippets you can use nosnippet and to disable caching use noarchive. Commented Aug 18, 2022 at 7:28
3

Nowadays, don't worry about it. Although we don't set the x-robot http header noindex for static file css, js, Googlebot know what file should be indexed or not. Except image, because the images can grab organic traffic for our site.

Btw, So many people still don't understand what is robots.txt file and noindex tag. Robots text file what should to crawl and what not to crawl, for indexing? back to our settings. While noindex tag is what should to index and what not to index.

Robots.txt file = Please come in my home, you can see this, this but not this, you dont allowed view this room.

Noindex tag (including x-robot http header): you may see the entire contents of this home but please don't open it to others (keep it secret) for this, this...

1
  • TLDR webmasters should stop using robots.txt so much, it can hurt your SEO and annoy the Google bots to prevent crawling of arbitrary content. On the other hand, the noindex header is becoming more relevant and useful. Commented Aug 18, 2022 at 7:30
1

if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right?

You are right, this is not good. Google knows very well how to use your CSS and JavaScript files to represent your site better in search results.

See this lesson with Matt Cutts about CSS, JavaScript, crawling and indexing: https://www.youtube.com/watch?v=m293K8JR2j4

0

My opinion on noindexing file types is that, generally speaking, trying to outsmart Google doesn't tend to work well and/or risks eventually becoming outdated as seen with @John's own answer.

To be more specific, file types that most websites are "expected" to be using in their web design should probably never be noindexed. Logically, it should be fine to noindex files like css or js or xml but realistically search engines like Google want to crawl those files, and trying to tell them exactly how to crawl them isn't going to do anything much besides risk annoying their bots, etc. Google's algorithm is incredibly intuitive, and they know which types of files are worth indexing more often than not.

However, I do think it makes sense to noindex certain types of files that are NOT typically part of your web design... files that may have inadvertantly been uploaded to your server over the years, or might be offered to users as a download, or things like that.

In particular, I think it often makes sense to noindex pdf and doc type files, among other textual files that almost never were uploaded as independent sources of information... the average website or blog only hosts those types of files as part of another web page or service, and if users come across those files while browsing search engines, the context of why the file exists is nearly always lost on them.

The below Nginx snippet will noindex most non-programming file types that Google claims to index:

## allowed: html, xml (also css, js, php, etc)
## Ref: https://support.google.com/webmasters/answer/35287?hl=en

location ~* \.(asp|aspx|doc|docx|dwf|gpx|htm|hwp|jsp|kml|kmz|log|odp|ods|odt|pdf|ppt|pptx|ps|rtf|tex|text|txt|wap|wml|xls|xlsx)$ {
    set $robots "noindex, nofollow, nosnippet, noarchive";
}

We are using this in SlickStack for cleaner SEO, along with noindexing some other regex patterns like WordPress pagination, tags, RSS feeds, etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.