1

have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't change it.

Edit: After a further research, I didn't find anything concrete about "index budget". Is there even such a thing? Can you confirm? Maybe I was wrong in my point of view.

But I read that using both "noindex" in meta tag and "dissallow" in robots.txt is not the best practice, or might be even harmful.

Do you know anything about this? How would you proceed?

Thank you!

PS: Tried to research this, read lots of articles and found a question as well (Internal Search Results: NOINDEX or robots.txt Blocking?), but didn't find an answer,

1 Answer 1

2

Crawl budget

If you have a very large number URLs, you should disallow them using robots.txt. This is especially the case if the content on those URLs is thin, unoriginal, machine generated, or otherwise low quality.

For most sites, Google is willing to crawl 100 URLs for every page that gets indexed. It is fairly rare for sites to have crawl budget problems unless they have very dynamic URLs.

The typical use cases for disallowing large numbers of URLs are when:

  • The URL space is arbitrary and infinite (ie. the user can type whatever they want.) For example search results.
  • There are different URLs for combinations of parameters. For example faceted navigation where shoppers choose product attributes to narrow a large product catalog.

When you disallow URLs, Googlebot won't crawl them and Google doesn't usually index them. Google only indexes URLs that it can't crawl when they seem important based on external links to them.

Index budget

It is very common for Google to choose to index only a small portion of the pages from a large site. However, I wouldn't call it an "index budget." It isn't like applying noindex to one page will make room for another page. Rather, Google makes the decision to index each page based on its own merits. That is:

  • The quality of the content. Good content is original, well written, and sufficiently covers the topic.
  • The reputation of the content. Google measures reputation by links.
  • Interest in the content. Google may decide not to index content that isn't getting used in the search results.

If only a small percentage of your pages are getting indexed, there are better strategies than pulling the pages that are currently indexed out of the index:

  • Give it time. Google automatically indexes more content year by year.
  • Improve the quality of your content.
  • Improve the internal link structure of your site to better highlight your best content.

For more information see Why aren't search engines indexing my content?

1
  • Amazing, thank you very much. Dec 5, 2022 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.