8

Google is executing the JavaScript on my password creator website and indexing the random passwords that get generated by the JavaScript.

The random text that is generated could look spammy to Google and I'm afraid that it will hurt rankings. It is text that only Googlebot will see (users see their own generated text) so in a sense it is cloaking. I use some common words as password suggestions and there is a chance they could come out in the order of a popular search phrase. Also, every time Googlebot renders the page, it will get different text on it, so my page will end up looking like it changes way more frequently than it actually does.

The passwords are also showing up in the search results in the text snippet under the link for normal searches. They look ugly there, so I'd also like to prevent Google from using the generated text for the search snippets.

How do I prevent Googlebot from indexing the password content on my site? I'd like to have the rest of the content on the page indexed.

Improve this question
5
  • 1
    I don't see why "random text" would be considered spammy? However, the "Common Words" column (below) might be - but taken in context I would expect Google to simply ignore it.
    – MrWhite
    May 20, 2015 at 20:36
  • Google can get upset when users search for things that aren't on the page but your site comes up anyway. I take your point that the random text is unlikely to be searched.
    – Stephen Ostermiller ♦
    May 20, 2015 at 20:57
  • 1
    Do users really see this content? I suspect you're trying to fix something that's not broken.
    – John Mueller
    May 27, 2015 at 8:44
  • This text had been coming up in the snippet in the search results even when searching for something like "password creator", so yes, users were seeing it.
    – Stephen Ostermiller ♦
    May 27, 2015 at 9:51
  • Aside from other aspects... this is actually great snippet for such service 😁.
    – Lukáš Řádek
    Jan 28 at 20:59

3 Answers 3

Reset to default
10

robots.txt can block JavaScript files from Googlebot. http://www.robotstxt.org/ has more information about how to construct a robots.txt file.

You could put your JavaScript that shows the password into an external JavaScript file (called showlists.js):

$(document).ready(function(){
    showLists();
});

Call that JavaScript file in the page head:

<script src="/showlists.js">

Then disallow it in robots.txt:

disallow: /showlists.js

Then Googlebot would be able to crawl the page, but would not see passwords rendered on the page.

This method is the Google approved way of blocking page elements from Googlebot that avoids cloaking. The disadvantage of this method is that it requires an external JavaScript file which can make the page slower to load.

You could test your robots.txt file with online testers such as http://tools.seobook.com/robots-txt/analyzer/

Improve this answer
0
3

I created a cloaking function in JavaScript:

function isBot(){
    return /bot|crawl|slurp|spider/i.test(navigator.userAgent)
}

Then I use that function to either show the passwords onload, or to show a message saying why no passwords were generated:

if (isBot()){
    $('#isbot').show();
} else {
    showLists();
}

Now when I use the "Fetch and Render" feature of "Fetch as Google" from webmaster tools I get the following.

This is how Googlebot saw the page:

This is how a visitor to your website would have seen the page:

This solution does involve showing different text to visitors than to Google and it could be considered to be a violation of the Google guidelines because it is technically cloaking.

Improve this answer
11
  • 1
    To be honest, I do not think Google will beat you up over this. The content is still available, however, the sensitive content that is not shown would not be of any value for search and mostly would not be recognized as terms when indexed anyway. The only partial exception is your common words column which may not make sense when indexed anyhow. I suspect that Google is not looking for explicit 1 to 1 matches especially now that so much content is automated or dynamic. I am rather sure that Google is looking for a majority match and some level of consistency which you have.
    – closetnoc
    May 20, 2015 at 18:44
  • 2
    As closetnoc states - I don't think the JS generated content is really an issue? Which is worse, hiding some "random content" (which doesn't actually mean anything to anyone) or deliberately cloaking? Another idea (to avoid the cloaking) could be to generate the content on a button click (which you are doing by the looks anyway).
    – MrWhite
    May 20, 2015 at 20:42
  • 1
    One of the features of this site is that the passwords come up automatically without you having to click a button. It makes the site one button click easier to use than the competition.
    – Stephen Ostermiller ♦
    May 20, 2015 at 20:58
  • 1
    Setting css :after from JavaScript looks pretty complex and I'd think Googlebot would still be able to see it. If you think it would work, I'd like to see an answer about it though.
    – Stephen Ostermiller ♦
    May 20, 2015 at 22:20
  • 1
    Don't use cloaking to try to get around this -- it's against Google's Webmaster Guidelines & makes maintenance a big hassle.
    – John Mueller
    May 27, 2015 at 8:42
2

Is there a way you can take advantage of iframes?

Google is able to just look at the source of an iframe, but it can't crawl an iframe directly into the parent page, as far as I know.

So, the iframe source should be noindex.

Improve this answer
6
  • That might be worth exploring, especially if I could dynamically create the iframe with JavaScript and add the noindex tag with JavaScript.
    – Stephen Ostermiller ♦
    May 23, 2015 at 10:33
  • I was thinking: implement iframes for your password generator section div in your original document (or for every single column: random, writable, etc.). Use them to call different webpages with password generator scripts, and these webpages should have noindex in their <head> tag, or just use robots.txt to prevent crawling. Maybe it's simpler and it may work.
    – lucgenti
    May 23, 2015 at 10:50
  • Might the iframe have accessibility issues? (Not sure what, just wondering.)
    – MrWhite
    May 24, 2015 at 22:25
  • Yes. In fact there are some actions to take, like implementing a title parameter for the tag, and writing text inside the iframe for people who cannot read the content (it would be like a <noframe> tag).
    – lucgenti
    May 25, 2015 at 7:47
  • What makes including an iframed file faster than including a javascript file? From my experience, iframe content does get indexed and can be considered part of the parent. You would still have to block it via robots.txt
    – Tony McCreath
    May 28, 2015 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.