3

I need to prevent robots from indexing specific word on my webpage.

Here is what I did:

Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the course of its first five years, JJJ was successful enough in providing shoes for children in need

Then I added this jQuery code to the page:

$("div.paragraph").each(function() {
    var text = $(this).text();
    newText = text.replace("JJJ", "JOHN GALLOW");
    if (text !== newText)
        $(this).text(newText);
});

That way, the right word will be injected at runtime.

Technically it works but I would like to know if this will be a good way for preventing robots from indexing.

  • 2
    Google process JavaScript so, no, this would not be a good way to prevent the indexing of this word – John Conde Mar 5 '16 at 15:16
  • So it seems to be a hard thing to do... – Bronzato Mar 5 '16 at 15:19
  • 1
    To see whether Google can read your hidden (with JavaScript) text you can use the "Fetch and Render" tool in Google Search Console (formerly GWT). – MrWhite Mar 6 '16 at 10:13
  • My question is why do you want to do this?? If you have made something public, should it not also be found by search? Otherwise, remove it. In effect, you are talking about sculpting search beyond the limits that Google appreciates. It just does not make sense to me. – closetnoc Mar 6 '16 at 16:14
-1

If the user-agent is Googlebot, then you can serve a different content, by hiding whatever you want.

if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot"))
{
    // remove the words
}
else
{
   //normal text
}

But Google will not like serving different content, but if the difference is minor, there shouldn't be any problem.

| improve this answer | |
  • So far it seems to be the best approach to serve my needs. But what do you mean by Google will not like serving different content ? Thanks anyway. – Bronzato Mar 6 '16 at 11:51
  • 2
    @Bronzato Sniffing the user-agent string and serving different content to search engines (ie. Google) than to normal users is known as "cloaking". This is something that Google could actively penalise. However, it's difficult to imagine that blocking a single word would have an effect on ranking - but it might. – MrWhite Mar 6 '16 at 17:19
1

Whilst I question your reason for this, one solution would be to use an image with the text. Something like:

<p>Over the course of its first five years, <img src="hiddentext.jpg">
was successful enough in providing shoes for children in need</p>

You'll need to be a bit creative with CSS if the site is responsive and you don't want it to be obvious to visitors.

| improve this answer | |
1

Google is pretty smart to execute JavaScript and CSS, but you can host your jQuery on a particular directory, for example http://example.com/js/jquery.min.js, and block that directory with robots.txt, so Google will not able to crawl that directory:

User-agent: Googlebot
Disallow: /js/

Here is to note that, you must be using jQuery to replace text only. If you do it for other purposes, then Google will not render other things, so host the jQuery file or host particular function to JS directory. I suggest to use Fetch and render tools from Google Webmaster Tools, to checkout how Google view your page.

Another thing is that, it might be treat as cloaking, but if you do for better UX and not for manipulating search ranking, then go ahead.

Update: If you did not have too much access, then you can use rel="nofollow" attribute in js src, like this<script rel="nofollow" src="/js/jquery.min.js"> Many of people think rel attribute is not supported by script tag, but Google matt cutts said in one of the video that, we also lookout rel attribute in script tag as well, they support it very well, you can use fetch and render tools to checkout preview.

| improve this answer | |
  • Thanks but unfortunately I don't have the possibility to host my jQuery to a particular directory... – Bronzato Mar 6 '16 at 11:23
  • Furthermore I don't have access to robots.txt (root) file. – Bronzato Mar 6 '16 at 11:39
  • 1
    Google's bot can occasionally visit your site unannounced without using a bot user agent, you should avoid any type of manipulation using user-agent. – Simon Hayter Mar 6 '16 at 16:09
  • Simon@Googlebot is politebot and all polite bot, strictly follow robots.txt guideline. Bronzato@Let me update my answer. – Goyllo Mar 7 '16 at 5:36
  • @Goyllo - Yes Googlebot follows the robots.txt guidelines, however they also have bots which use standard browser user agent strings and in some instances don't even appear to use Google assigned IP addresses. These bots are used specifically for identifying attempts at claoking so that the appropriate warnings and possibly penalties can be applied. As these bots are not directly related to the action of indexing for the search engine and are instead a verification toolset they don't have to follow the robots.txt guideline and trying to force them to would defeat their purpose. – Chris Rutherfurd Jan 27 '17 at 8:55
1

Sometimes I do this by hosting an extra webfont copy with letters that change places so that the word becomes senseless.

This in theory does not prevent it from being indexed but not in a meaningful context, so this serves for its purpose. Obviously you need to be familiar with font creating and host it by yourself.

You could move letters one glyph forwards, so you have A for B, B for C etc. Then create something like

<span class="no-follow">Qerklzb BcjrzfXzhc</span>

Attach the new font to the class. That’s all about it.

| improve this answer | |
0

While there have been many answers providing methods on how this could theoretically can be done all of these methods have been discovered by Google and have been made ineffective. Google is able to parse CSS and javascript to identify what it is trying to do to the page, as well as being able to identify text that appears in images.

Basically you are unable to have Google from indexing a specific word in the page. The key point is that Google will index all significant words in your page but how they rank in the Google SERP is what matters. Rather than trying to prevent Google from indexing certain words which will by any technique result in Google deeming it cloaking, instead you should focus on trying to increase the relative value of the other words you do want indexed to increase the ranking of your site based on those keywords.

| improve this answer | |
-1

If you are concerned about bots reading your JavaScript code you can use some simple tricks to confuse them. For example, convert the text you don't want them to understand in an array of integers and then use that array to rebuild the text at runtime like you are doing. You can use the following JavaScript function to turn texto into an array of integers offline:

function text_to_array(txt){
  var i = 0 , x = [];
  for(i = 0; i < txt.length; i++){
    x[x.length] = String.charCodeAt(txt[i]);
  }
  return x;
}
alert(text_to_array("JOHN GALLOW"));

And that would give you a message with the array. In the JavaScript code of your page use the following code.

function array_to_text(arr){
  var i = 0 , x = "";
  for(i = 0; i < arr.length; i++){
    x = x + String.fromCharCode(arr[i]);
  }
  return x;
}
TXT = [74,79,72,78,32,71,65,76,76,79,87];
  /* Just replaced in your code, haven't checked it */
$("div.paragraph").each(function() {
  var text = $(this).text();
  newText = text.replace("JJJ", array_to_text(TXT));
  if (text !== newText)
    $(this).text(newText);
});
| improve this answer | |
  • How is this method different from mine ? I mean is this because the words JOHN GALLOW are not explicitly present in the javascript code and then Google is not able to index it ? Thanks anyway. – Bronzato Mar 6 '16 at 8:38
  • 2
    Obfuscating the text with code that is hard to read for humans doesn't necessarily make this any harder for a bot that uses a JavaScript interpreter, ie. Google. – MrWhite Mar 6 '16 at 10:16
-2

There are some HTML tags to block Google indexing special parts of your page.

googleoff googleon

You can refer to the below article which helps you more understand the concept https://perishablepress.com/tell-google-to-not-index-certain-parts-of-your-page/

| improve this answer | |
  • 2
    googleoff and googleon applies only to "Google Search Appliance" - this is not the "Google" that everyone uses to search the web. (This correction is made in the article you link to.) – MrWhite Mar 5 '16 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.