Crawl budget
If you have a very large number URLs, you should disallow them using robots.txt. This is especially the case if the content on those URLs is thin, unoriginal, machine generated, or otherwise low quality.
For most sites, Google is willing to crawl 100 URLs for every page that gets indexed. It is fairly rare for sites to have crawl budget problems unless they have very dynamic URLs.
The typical use cases for disallowing large numbers of URLs are when:
- The URL space is arbitrary and infinite (ie. the user can type whatever they want.) For example search results.
- There are different URLs for combinations of parameters. For example faceted navigation where shoppers choose product attributes to narrow a large product catalog.
When you disallow URLs, Googlebot won't crawl them and Google doesn't usually index them. Google only indexes URLs that it can't crawl when they seem important based on external links to them.
Index budget
It is very common for Google to choose to index only a small portion of the pages from a large site. However, I wouldn't call it an "index budget." It isn't like applying noindex to one page will make room for another page. Rather, Google makes the decision to index each page based on its own merits. That is:
- The quality of the content. Good content is original, well written, and sufficiently covers the topic.
- The reputation of the content. Google measures reputation by links.
- Interest in the content. Google may decide not to index content that isn't getting used in the search results.
If only a small percentage of your pages are getting indexed, there are better strategies than pulling the pages that are currently indexed out of the index:
- Give it time. Google automatically indexes more content year by year.
- Improve the quality of your content.
- Improve the internal link structure of your site to better highlight your best content.
For more information see Why aren't search engines indexing my content?
