Should tag and author pages be disallowed in robots.txt?

Question

According to some articles that talk about crawl budget, we should decrease the number pages crawled and provide the only important and valuable pages to search engine crawlers.

We have 3,000 articles on our web site but the number of indexed pages in Google Search Console is 12,000. The difference between 12,000 and 3,000 is our tag pages and author pages.

What should I do with tags and authors page? Should I block these pages? If so, where would the block be implemented? In robots.txt or meta tag in headers?

Answer 1

Very few sites are limited by Googlebot's crawl budget. Googlebot is usually willing to crawl 10 times as many pages at it is willing to index. I wouldn't block Googlebot from crawling those pages for crawl budget reasons. If you have 12,000 pages indexed you could probably have 120,000 pages on your site available for Googlebot to crawl before you ran into any crawl budget problems.

There are pluses and minuses to tag and author pages:

• Plus: They tie your site together from a link graph standpoint.
• Plus: They provide landing pages for people search "category" type search terms.
• Minus: They are often thin content.

Your site appears to be doing very well at getting your content indexed. You probably shouldn't change anything.

If your were to prevent them from being indexed, I would start with the worst ones. Use a meta noindex tag on author pages that haven't filled in any information. Do the same for tag pages that only list a couple articles.