Possible Duplicate:
What is duplicate content and how can I avoid being penalized for it on my site?

I'm hoping that someone has a solution for what I'm trying to accomplish. I'm working on a travel agency web site and there's a "Overview" section for each cruise line. These overviews are located on the index page for each cruise line.

Here's my issue: The company is creating a search engine that includes details on each cruise line. Their write-ups on each cruise line are great, so I'd like to include the overview they created for each cruise line, rather than having to create all new ones. However, I don't want duplicating their content to negatively affect the SEO scores of the pages they originally put this content on. It's gong to duplicate, since each page that's dynamically generated by their search engine is going to include a section about the cruise line (where I'd want to place the overview).

Question: Is there any way that I can include these overviews (ideally, copying the exact HTML that they've already implemented) without the search engines indexing those particular code sections? I'd want the rest of the search result pages to be indexed...just not the section of each page that contains this duplicate code.

I saw something about using a span class named robots-nocontent in Yahoo (not sure if this also applies to Bing) and googleon / googleoff tags in Google. Is this the best solution?

I'm open to any suggestions, thanks!

  • I'm not necessarily saying your idea is good or bad but if you add this content with JavaScript when the page is loaded (for example) then GoogleBot (etc.) won't see it as its not actually in the markup of the page – Adam Lynch Sep 24 '12 at 18:13

Short Answer:
No, there's no way to markup a piece of content on a page that blocks or excludes that content from being crawled/indexed.

Longer Answer:
I think you're being overly concerned about duplicate content and how it is determined for the purposes of SEO. In the eyes of the search engines, duplicate content refers to

"substantive blocks of content within or across domains that either completely match other content or are appreciably similar".

Specifically, in your case, you have less to worry about because you are legitimately trying to add consumer value by providing information about each cruise. This piece from Google explains it well:

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results.

So, unless those cruise line descriptions make up the substantive part of the content you have little to worry about.

There's more information on understanding and managing duplicate content here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66359


It is well-known to search engines that onsite - search result pages contain snippets of the content that is searched for.


I would recommend you set the values "noindex,follow" for your robot-tag on your dynamically created search result pages.

meta name="robots" content="noindex,follow"

This way the results don't accidentally compete with your main content and any search engine will still follow all links and spider your site. The robots tag is known to all bots.

  • 2
    Good answer. Search engines results pages should NOT be indexed. – DisgruntledGoat Sep 24 '12 at 11:42
  • @DisgruntledGoat there are good exceptions - real estate listings result pages, car for sale listings etc. - anything really that falls under the 'classifieds' label. – Mike Hudson Sep 25 '12 at 2:20
  • @MikeHudson But then - for those to rank well it might often be better to keep the results more predictable, more static, with less variations (like with simple categories and pagination) to avoid looking spammy. – initall Sep 25 '12 at 6:55
  • @Mike I wouldn't really class those as search results. Initall is right - using categories/filters is a better option for search engines to index. – DisgruntledGoat Sep 25 '12 at 19:49

Generally speaking, our (I work at Google on web-search) algorithms are really good at recognizing that kind of duplication and treating it appropriately. You don't need to do anything special in a case like that. In particular, you might see that only one of the pages that contains the exact same content is showing in the search results (that's not a penalty, it just helps with the user-experience since it doesn't make sense to show the same content multiple times in the search results).

If you wanted to exclude certain content within a page from being indexed (and again, in most cases you don't need to worry about this!), one way you could do it is to include that content via an iframe, and to use the robots.txt file to prevent crawling of that iframe'd content. That way, search engines will be able to crawl the page, but won't be able to fetch the content that's embedded via iframe. Since our systems are generally really good at handling duplicate parts of content like that, personally I wouldn't bother doing that, but in the end, it's up to you.


You could look into "cross-domain canonical links," and this Spotted Panda article touches on this topic in the fourth bullet point in 'on-page' factors.

In short (and this changes so quickly I could be mistaken), you probably won't be 'penalized' (as in harmed), but search engines (read as: Google) try not to show duplicate content in SERPs.

Not the answer you're looking for? Browse other questions tagged or ask your own question.