Skip to main content
Tweeted twitter.com/#!/StackWebmasters/status/502993827976998912
Post Reopened by dan
added 727 characters in body
Source Link
WPRookie82
  • 443
  • 2
  • 8

I am working on a simple Wordpress blog where when an article is published, it appears on ALL these pages:

  1. Homepage - Headline(clickable) + 3-line summary

  2. Parent category page - Headline(clickable) + 3-line summary

  3. Child category page - Headline(clickable) + 3-line summary

  4. Author page - Headline(clickable)

  5. sitemap.xml

A good example of what I mean is:

http://wp-themes.com/twentyfourteen/

The "Worth A Thousand Words" article is available on the homepage.

But it's also available on the category page: http://wp-themes.com/?cat=1

And finally, it's also available on the author apge: http://wp-themes.com/?author=1

Does it makes sense to let Google crawl http://wp-themes.com/?author=1 knowing that all links on that page were already crawled elsewhere?

I've been in fact told that I should add all author pages to my robots.txt, under disallow, so as search engine bots do not spider /author/*(or anything with ?author for the above example) since all links on these pages are indeed available elsewhere.

Is this a good approach or maybe rel=nofollow is better, or maybe I shouldn't worry about this at all?

I am working on a simple Wordpress blog where when an article is published, it appears on ALL these pages:

  1. Homepage - Headline(clickable) + 3-line summary

  2. Parent category page - Headline(clickable) + 3-line summary

  3. Child category page - Headline(clickable) + 3-line summary

  4. Author page - Headline(clickable)

  5. sitemap.xml

I've been told that I should add all author pages to my robots.txt, under disallow, so as search engine bots do not spider /author/* since all links on these pages are available elsewhere.

Is this a good approach or maybe rel=nofollow is better, or maybe I shouldn't worry about this at all?

I am working on a simple Wordpress blog where when an article is published, it appears on ALL these pages:

  1. Homepage - Headline(clickable) + 3-line summary

  2. Parent category page - Headline(clickable) + 3-line summary

  3. Child category page - Headline(clickable) + 3-line summary

  4. Author page - Headline(clickable)

  5. sitemap.xml

A good example of what I mean is:

http://wp-themes.com/twentyfourteen/

The "Worth A Thousand Words" article is available on the homepage.

But it's also available on the category page: http://wp-themes.com/?cat=1

And finally, it's also available on the author apge: http://wp-themes.com/?author=1

Does it makes sense to let Google crawl http://wp-themes.com/?author=1 knowing that all links on that page were already crawled elsewhere?

I've been in fact told that I should add all author pages to my robots.txt, under disallow, so as search engine bots do not spider /author/*(or anything with ?author for the above example) since all links on these pages are indeed available elsewhere.

Is this a good approach or maybe rel=nofollow is better, or maybe I shouldn't worry about this at all?

Post Closed as "Needs details or clarity" by John Conde
Source Link
WPRookie82
  • 443
  • 2
  • 8

Should I disallow(robots.txt) archive/author pages with links already available on the front page?

I am working on a simple Wordpress blog where when an article is published, it appears on ALL these pages:

  1. Homepage - Headline(clickable) + 3-line summary

  2. Parent category page - Headline(clickable) + 3-line summary

  3. Child category page - Headline(clickable) + 3-line summary

  4. Author page - Headline(clickable)

  5. sitemap.xml

I've been told that I should add all author pages to my robots.txt, under disallow, so as search engine bots do not spider /author/* since all links on these pages are available elsewhere.

Is this a good approach or maybe rel=nofollow is better, or maybe I shouldn't worry about this at all?