1

We have our own internal search results pages for our download section.

These results are currently crawlable but with NOINDEX meta tags so they don't appear in the search index.

People link to these results - but we have 10s of thousands of pages of these results, possibly eating up lots of crawl budget.

What should we do - should we disallow indexing using robots.txt to these pages or should we continue with the META NOINDEX (follow) so people linking to these search result pages still pass their "link juice"?

Google states (in their webmaster guidelines): "Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages."

Any advice, suggestions or opinions are greatly appreciated. Thank you in advance.

  • Are the pages, that are linked to from the SERPs, crawlable in other ways? – DocRoot Jul 13 '17 at 22:08
  • 1
    No, they are not linked to internally - only external links to them from sites such as forums... etc. – Ted Wilmont Jul 15 '17 at 18:51
1

Interesting question! To me it really seems like a matter of crawl budget vs. link juice.

  • Is it perhaps possible to block certain parts of the result pages set? For instance those with particular parameters that nobody links to?
  • Does it seem like you're already having an issue with the crawl budget on your site? If not, then keeping the link juice might be a good solution instead.
  • Do these links come from high-quality websites? Is the link juice worth the dent in your crawling budget?
  • Are these search result pages optimized for a (set of) keyword(s)? While the increased link juice to your domain is positive, will/does it also lead to the search result pages getting traffic and/or generating conversions?
  • Do these search result pages contain fairly unique content, or is it mostly the-same-search-results-in-another-order?
  • Can you perhaps take the most-linked search result pages and create a good resource for those instead on your website, and then request the webmasters to change their links to the superior page?

As you can see, unfortunately there doesn't seem to be a simple solution here, but it comes down to the details.

  • But the page being linked to is noindex, so why would "link juice" even apply here? – DocRoot Jul 13 '17 at 22:28
  • NOINDEX keeps the page out of the index while allowing link juice to that page to be passed to all of the other links on the page. – Ted Wilmont Jul 14 '17 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.