I have a site where the same content can be found in different ways, for example from a search.

The canonical URL is always a URL with 'product-item', but other ways [to get to same content] might end up as a URL with 'search-item' (even though it has the product-item canonical URL).

Should I exclude search-item (say, in robots.txt) so bots don't crawl that potential dupe content, and also exclude from sitemap files? Or does having the canonical URL take care of this?

  • Canonical tags are the way to go. For useless pages that should not be indexed, it is more common to use noindex on the page. If these pages can be specified in your robots.txt, then you can do it, however, I would advise noindex instead. Just a personal preference. – closetnoc Jul 26 '16 at 0:59

Yes, you should exclude the duplicate content from sitemap, you need to add only your preferred url for duplicate webpages, as Google says:

Pick a canonical (preferred) URL for each of your pages, and tell us about your preference by submitting these canonical URLs in a sitemap.

We don't guarantee that we'll use the URLs you submit in a sitemap, but submitting one is a useful way to tell Google about the pages on your site you consider most important.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.