3

I need to prevent robots from indexing specific word on my webpage.

Here is what I did:

Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the course of its first five years, JJJ was successful enough in providing shoes for children in need

Then I added this jQuery code to the page:

$("div.paragraph").each(function() {
    var text = $(this).text();
    newText = text.replace("JJJ", "JOHN GALLOW");
    if (text !== newText)
        $(this).text(newText);
});

That way, the right word will be injected at runtime.

Technically it works but I would like to know if this will be a good way for preventing robots from indexing.

4
  • 2
    Google process JavaScript so, no, this would not be a good way to prevent the indexing of this word
    – John Conde
    Mar 5, 2016 at 15:16
  • So it seems to be a hard thing to do...
    – Bronzato
    Mar 5, 2016 at 15:19
  • 1
    To see whether Google can read your hidden (with JavaScript) text you can use the "Fetch and Render" tool in Google Search Console (formerly GWT).
    – MrWhite
    Mar 6, 2016 at 10:13
  • My question is why do you want to do this?? If you have made something public, should it not also be found by search? Otherwise, remove it. In effect, you are talking about sculpting search beyond the limits that Google appreciates. It just does not make sense to me.
    – closetnoc
    Mar 6, 2016 at 16:14

7 Answers 7

1

Whilst I question your reason for this, one solution would be to use an image with the text. Something like:

<p>Over the course of its first five years, <img src="hiddentext.jpg">
was successful enough in providing shoes for children in need</p>

You'll need to be a bit creative with CSS if the site is responsive and you don't want it to be obvious to visitors.

1

Google is pretty smart to execute JavaScript and CSS, but you can host your jQuery on a particular directory, for example http://example.com/js/jquery.min.js, and block that directory with robots.txt, so Google will not able to crawl that directory:

User-agent: Googlebot
Disallow: /js/

Here is to note that, you must be using jQuery to replace text only. If you do it for other purposes, then Google will not render other things, so host the jQuery file or host particular function to JS directory. I suggest to use Fetch and render tools from Google Webmaster Tools, to checkout how Google view your page.

Another thing is that, it might be treat as cloaking, but if you do for better UX and not for manipulating search ranking, then go ahead.

Update: If you did not have too much access, then you can use rel="nofollow" attribute in js src, like this<script rel="nofollow" src="/js/jquery.min.js"> Many of people think rel attribute is not supported by script tag, but Google matt cutts said in one of the video that, we also lookout rel attribute in script tag as well, they support it very well, you can use fetch and render tools to checkout preview.

6
  • Thanks but unfortunately I don't have the possibility to host my jQuery to a particular directory...
    – Bronzato
    Mar 6, 2016 at 11:23
  • Furthermore I don't have access to robots.txt (root) file.
    – Bronzato
    Mar 6, 2016 at 11:39
  • 1
    Google's bot can occasionally visit your site unannounced without using a bot user agent, you should avoid any type of manipulation using user-agent. Mar 6, 2016 at 16:09
  • Simon@Googlebot is politebot and all polite bot, strictly follow robots.txt guideline. Bronzato@Let me update my answer.
    – Goyllo
    Mar 7, 2016 at 5:36
  • @Goyllo - Yes Googlebot follows the robots.txt guidelines, however they also have bots which use standard browser user agent strings and in some instances don't even appear to use Google assigned IP addresses. These bots are used specifically for identifying attempts at claoking so that the appropriate warnings and possibly penalties can be applied. As these bots are not directly related to the action of indexing for the search engine and are instead a verification toolset they don't have to follow the robots.txt guideline and trying to force them to would defeat their purpose. Jan 27, 2017 at 8:55
1

Sometimes I do this by hosting an extra webfont copy with letters that change places so that the word becomes senseless.

This in theory does not prevent it from being indexed but not in a meaningful context, so this serves for its purpose. Obviously you need to be familiar with font creating and host it by yourself.

You could move letters one glyph forwards, so you have A for B, B for C etc. Then create something like

<span class="no-follow">Qerklzb BcjrzfXzhc</span>

Attach the new font to the class. That’s all about it.

0

While there have been many answers providing methods on how this could theoretically can be done all of these methods have been discovered by Google and have been made ineffective. Google is able to parse CSS and javascript to identify what it is trying to do to the page, as well as being able to identify text that appears in images.

Basically you are unable to have Google from indexing a specific word in the page. The key point is that Google will index all significant words in your page but how they rank in the Google SERP is what matters. Rather than trying to prevent Google from indexing certain words which will by any technique result in Google deeming it cloaking, instead you should focus on trying to increase the relative value of the other words you do want indexed to increase the ranking of your site based on those keywords.

-1

If the user-agent is Googlebot, then you can serve a different content, by hiding whatever you want.

if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot"))
{
    // remove the words
}
else
{
   //normal text
}

But Google will not like serving different content, but if the difference is minor, there shouldn't be any problem.

2
  • So far it seems to be the best approach to serve my needs. But what do you mean by Google will not like serving different content ? Thanks anyway.
    – Bronzato
    Mar 6, 2016 at 11:51
  • 2
    @Bronzato Sniffing the user-agent string and serving different content to search engines (ie. Google) than to normal users is known as "cloaking". This is something that Google could actively penalise. However, it's difficult to imagine that blocking a single word would have an effect on ranking - but it might.
    – MrWhite
    Mar 6, 2016 at 17:19
-1

If you are concerned about bots reading your JavaScript code you can use some simple tricks to confuse them. For example, convert the text you don't want them to understand in an array of integers and then use that array to rebuild the text at runtime like you are doing. You can use the following JavaScript function to turn texto into an array of integers offline:

function text_to_array(txt){
  var i = 0 , x = [];
  for(i = 0; i < txt.length; i++){
    x[x.length] = String.charCodeAt(txt[i]);
  }
  return x;
}
alert(text_to_array("JOHN GALLOW"));

And that would give you a message with the array. In the JavaScript code of your page use the following code.

function array_to_text(arr){
  var i = 0 , x = "";
  for(i = 0; i < arr.length; i++){
    x = x + String.fromCharCode(arr[i]);
  }
  return x;
}
TXT = [74,79,72,78,32,71,65,76,76,79,87];
  /* Just replaced in your code, haven't checked it */
$("div.paragraph").each(function() {
  var text = $(this).text();
  newText = text.replace("JJJ", array_to_text(TXT));
  if (text !== newText)
    $(this).text(newText);
});
2
  • How is this method different from mine ? I mean is this because the words JOHN GALLOW are not explicitly present in the javascript code and then Google is not able to index it ? Thanks anyway.
    – Bronzato
    Mar 6, 2016 at 8:38
  • 2
    Obfuscating the text with code that is hard to read for humans doesn't necessarily make this any harder for a bot that uses a JavaScript interpreter, ie. Google.
    – MrWhite
    Mar 6, 2016 at 10:16
-2

There are some HTML tags to block Google indexing special parts of your page.

googleoff googleon

You can refer to the below article which helps you more understand the concept https://perishablepress.com/tell-google-to-not-index-certain-parts-of-your-page/

1
  • 2
    googleoff and googleon applies only to "Google Search Appliance" - this is not the "Google" that everyone uses to search the web. (This correction is made in the article you link to.)
    – MrWhite
    Mar 5, 2016 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.