Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally polluted by my users' signatures (they're using long signatures and that's part of the forum's experience because signatures makes a lot of sense in my forum).

So basically I'm seeing two options as of now:

  1. Rendering the signature as a picture and when a user click on the "signature picture" it gets taken to a page that contains the real signature (with the links in the signature etc.) and that page is set as being non-crawlable by search engine spiders). This would consume some bandwidth and need some work (because I'd need an HTML renderer producing the picture etc.) but obviously it would solve the issue (there are tiny gotchas in that the signature wouldn't respect the font/color scheme of the users but my users are very creative with their signatures anyway, using custom fonts/colors/size etc. so it's not that much of an issue).

  2. Marking every part of the webpage that contains a signature as being non-crawlable.

However I'm not sure about the later: is this something that can be done? Can you just mark specific parts of a webpage as being non-crawlable?

share|improve this question

4 Answers 4

Here is the same answer I provided to noindex tag for google on Stack Overflow:

You can prevent Google from seeing portions of the page by putting those portions in iframes that are blocked by robots.txt.

robots.txt

Disallow: /iframes/

index.html

This text is crawlable, but now you'll see 
text that search engines can't see:
<iframe src="/iframes/hidden.html" width="100%" height=300 scrolling=no>

/iframes/hidden.html

Search engines cannot see this text.

Instead of using using iframes, you could load the contents of the hidden file using AJAX. Here is an example that uses jquery ajax to do so:

his text is crawlable, but now you'll see 
text that search engines can't see:
<div id="hidden"></div>
<script>
    $.get(
        "/iframes/hidden.html",
        function(data){$('#hidden').html(data)},
    );
</script>
share|improve this answer

This is easy.

Before you serve your page you need to know whether it is to a bot, a computer or a phone. You then need to set the content accordingly. This is standard practice in this day and age and core functionality of some CMS's.

There are plenty of solutions on SE for doing redirection based on USER AGENT that can be put in your htaccess. If this suits your forum software then you can run different code off the same DB to deliver what Google needs without the chaff and trimmings.

Alternatively you can put a little line in your PHP code that does a 'if USER AGENT == Googlebot then don't show signatures'.

If you really cannot do that then you can get mod_proxy to serve to the bot and use it to strip out anything your php code generates that the bot not need to see.

Technically Google do not approve of their search engine being shown a different page to what the normal site visitor sees, however, to date, they have not taken the BBC and others that provide browser/IP/visitor-specific content off their search engine results. They also have limited means to see if their bot has been 'conned'.

The alternative solution of hiding content with CSS for it to be re-enabled by a script is also a bit of a grey area. According to their own Webmaster Tools guidelines of 20/6/11 this is not a good idea:

http://www.google.com/support/webmasters/bin/answer.py?answer=66353

That may not be a tablet cast in stone, but it is up to date and by Google.

The hide the content trick will not work with the minority of people that do not have javascript, this may not be a huge concern, however, waiting for the document to load and then showing the signatures will not be a satisfactory viewing experience as you will think the page has loaded, then it will jump about as the hidden signatures show up to then push the content down the page. This type of page load can be irritating if you have a low-end net-top but may not be noticeable if you have a fast developers machine on a fast internet connection.

share|improve this answer
1  
@ʍǝɥʇɐɯ: serving different content depending on who is accessing the page is kinda frowned upon and may penalize you in search engine as far as I understand it. I much prefer paulmorris' JavaScript solution. –  WebbyTheWebbor Jul 4 '11 at 15:34
    
@ʍǝɥʇɐɯ: erf, if serving personalized content is the name of the game, so is JavaScript. Last I checked the Web overall didn't really work that well anymore without JavaScript installed (GMail, FaceBook, Google Docs, stack overflow, Google+ --yup I've got it already ;) -- etc.). I don't see no need to criticize paulmorris' solution based on the false premise that JavaScript being not available would be an issue. –  WebbyTheWebbor Jul 4 '11 at 16:19
    
@ʍǝɥʇɐɯ: You may like this from Matt Cutts (in charge of SEO at Google) on that very subject: theseonewsblog.com/3383/google-hidden-text That was the excellent comment by paulmorris posted in comment to his excellent answer. I'm sorry but calling JavaScript "sillyness" on such a forum is close to trolling. –  WebbyTheWebbor Jul 4 '11 at 19:34
    
...and then we get this question: webmasters.stackexchange.com/questions/16398/… - 'keyword stuffing' is silly. Sorry about that. –  ʍǝɥʇɐɯ Jul 4 '11 at 20:17

No, there is no way to prevent robots crawling parts of pages. It's a whole page or nothing.

The snippets in Google's search results are usually taken from the meta description on the page. So you could make Google show a specific part of the page by putting that in the meta description tag. With user-generated content it's difficult to get good snippets, but taking the first post of the thread would probably work.

The only other way I can think of is to use Javascript. Something like paulmorriss suggested may work, but I think search engines would still index the content if it's in the HTML. You could remove it from the HTML, store it in Javascript string, then add it back on page load. This gets a bit complex, though.

Finally, one thing to keep in mind: if Google is showing user's signatures in their snippets, it has decided that is the part most relevant to the user's query.

share|improve this answer
    
the problem is not so much Google showing user's sigs in their snippets as these specific pages getting that highly ranked in Google in the first place. The issue here is precisely that Google may think the sigs are relevant when they're actually not: I mean, that's exactly what my question is all about. –  WebbyTheWebbor Jul 4 '11 at 15:32
    
@Webby, I don't understand, why don't you want your pages ranking highly? Do you have some example pages and queries so we can see what you're talking about? And if Google is showing a sig in search results, then it is relevant for that search query, even if it's not relevant to the page itself. –  DisgruntledGoat Jul 4 '11 at 16:05
    
I can't give examples but I do want my site/forum to rank highly and it does so very nicely. The problem is that amongst the search results (which are all mostly for my site/forum anyway because it's basically the site on the subject), what should be the real entry pages are flooded amongst signatures. I mean, I do really want to do what I asked in the question. And pictures or JavaScript it is going to be. –  WebbyTheWebbor Jul 4 '11 at 16:24
    
@Webby, your responses have been a little confusing but you seem to be implying that your user signatures are all separate pages (URLs) and thus appearing as separate results in SERPs. In which case you can block those pages through robots.txt. Otherwise, try the meta description solution I posted above, because that will almost certainly mitigate the problem. –  DisgruntledGoat Jul 5 '11 at 9:42

Another solution is to wrap the sig in a span or div with style set to display:none and then use Javascript to take that away so the text displays for browsers with Javascript on. Search engines know it's not going to be displayed so shouldn't index it.

This bit of HTML, CSS and javascript should do it:

HTML:

<span class="sig">signature goes here</span>

CSS:

.sig {
display:none;
}

javascript:

<script type="text/javascript"> 
$(document).ready(function()
  {
      $(".sig").show();
  }
</script>

You'll need to include a jquery library.

share|improve this answer
1  
+1 and I thought about it but wouldn't that be considered a form of "cloaking" by various spiders? –  WebbyTheWebbor Jul 4 '11 at 13:55
1  
Not by Google: theseonewsblog.com/3383/google-hidden-text –  paulmorriss Jul 4 '11 at 14:31
1  
I think it's quite neat :-) –  paulmorriss Jul 4 '11 at 14:42
    
This could, in the strictest definition, be considered cloaking. However he could print all of the signature with javascript using a document.write("");. Google does not index anything within javascript. support.google.com/customsearch/bin/… –  Patrik Alienus Jan 9 '13 at 15:33
    
I believe Google could index such paragraphs, even if they are hidden using CSS. The safest option is to not include the text in the HTML at all. (We can use JavaScript to inject the text at runtime.) –  wrygiel Sep 23 '13 at 15:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.