Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
Google says that it is effective to prevent Google Crawling Bot from crawling some useless pages in my site.
https://developers.google.com/search/docs/advanced/guidelines/how-search-works https://developers.google.com/search/docs/advanced/crawling/block-indexing
But I could not figure it out. Why is it effective to prevent Google Crawling Bot from crawling some useless pages to enhance SEO?
Some pages can hurt the SEO of your entire site if you let Google crawl and index them. From my answer to How should I choose which URLs I want indexed to include in my sitemap? there are several types of pages that Google doesn't want to index:
For each of those, you actually have four options:
robots.txtnoindex meta tag or headercanonical meta tagsGoogle will never index error pages or redirects so it is fine to let Googlebot crawl them decide not to index them.
For duplicate content it is usually best to redirect or tell Googlebot about the duplication with canonical tags.
I recommend preventing Googlebot from crawling very large (or infinite) URL spaces.
For other types of content you have to strike a balance between allowing lots of crawling, vs preventing indexing. Googlebot may occasionally index pages that it can't crawl, but to see the noindex Googlebot has to be able to crawl the URL.
If your site has only rich, original, informative URLs, then you can let Googlebot crawl the whole thing and index all of it.
X-Robots-Tag in addition to the 4 listed methods. Isn't another thing not to "use too much crawl power"? I think I read somewhere (citation needed) that if you say have 100.000 URLs, Google will somewhat rate limit how many of those are how often crawled. So excluding some could mean others are crawled more often.
X-Robts-Tag: noindex HTTP header is equivalent to the <meta name="robots" content="noindex"> tag. I edited the answer to say that either would prevent indexing.
Sep 7, 2021 at 9:25
robots.txt for infinite URL spaces. In general, Googlebot is willing to do a lot of crawling. Googlebot is usually willing to crawl 10 or 100 URLs for every real page. Your actual crawl budget is determined by PageRank. The more links into your site, the more Googlebot is willing to crawl.
Sep 7, 2021 at 9:25
Some pages may have special uses that you don't need Google to display in the search results.
For example, maybe you have a series of pages that asks the user for data in sequential steps. The first page asks for the user's name and maybe a password to complete an order. The second page asks for mailing address to complete the order.
While the first page is important for users to find in order to place an order, the second page is irrelevant without the first page. So, you add the meta tag <meta name="robots" content="noindex"> to the second page, preventing Google from showing the second page in it's search results.
A common black-hat SEO technique is to generate many, many pages designed to seem relevant to search engines, but actually full of machine-generated crud. The idea is for the page to "look good to Google" and hit #1 in search results, but then, guide the user to some other sales page to try to sell them something.
There is a huge arms race in search, between SEOs using these black-hat techniques and search engineers trying to stop those tricks from working and punish the sites that use them.
Google's methods are very heavy on algorithmic methods to find these sites, which analyze sites on a "site by site" basis. This has less human intervention than competitors... the algorithm doesn't overlook sites, but it's also more prone to false positives.
For instance, let's say you have an internal site search, which creates a URL like
http://www.example.com/search.cgi/search+term+here
Every time someone does a search, it creates a URL like this. And these URLs wind up in your web log analysis pages which Google can see for some reason. So there are 500,000 URLs like the above, simply from 500,000 different searches people have done.
Every one of these pages has the internal search results from within your site, with the search term repeated many times in the various snippets and titles.
Compare this to the black-hat SEO technique. The pages are machine generated, stuffed with the main keyword, and don't have original content. They look exactly like black-hat doorway pages! A human curator can see it for what it is, but an algorithm can't.
So you want to exclude that "directory" so you don't get mistaken for a black-hat spammer.
First of all, you should know that for SEO it is better to prevent indexing, not crawling. If you prevent crawling with robots.txt, pages can be indexed, even if you used meta tag or HTTP-header noindex. And of course, if it is appropriate, use the tag canonical.
You asked Why?. I know at least two reasons:
