Internal Search Results: NOINDEX or robots.txt Blocking?

Question

We have our own internal search results pages for our download section.

These results are currently crawlable but with NOINDEX meta tags so they don't appear in the search index.

People link to these results - but we have 10s of thousands of pages of these results, possibly eating up lots of crawl budget.

What should we do - should we disallow indexing using robots.txt to these pages or should we continue with the META NOINDEX (follow) so people linking to these search result pages still pass their "link juice"?

Google states (in their webmaster guidelines): "Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages."

Any advice, suggestions or opinions are greatly appreciated. Thank you in advance.

Answer 1

Interesting question! To me it really seems like a matter of crawl budget vs. link juice.

• Is it perhaps possible to block certain parts of the result pages set? For instance those with particular parameters that nobody links to?
• Does it seem like you're already having an issue with the crawl budget on your site? If not, then keeping the link juice might be a good solution instead.
• Do these links come from high-quality websites? Is the link juice worth the dent in your crawling budget?
• Are these search result pages optimized for a (set of) keyword(s)? While the increased link juice to your domain is positive, will/does it also lead to the search result pages getting traffic and/or generating conversions?
• Do these search result pages contain fairly unique content, or is it mostly the-same-search-results-in-another-order?
• Can you perhaps take the most-linked search result pages and create a good resource for those instead on your website, and then request the webmasters to change their links to the superior page?

As you can see, unfortunately there doesn't seem to be a simple solution here, but it comes down to the details.