I'm getting the following warning over and over again in google webmaster tools

Googlebot encountered extremely large numbers of links on your site.

The examples he's showing don't give me much clue on what is really wrong here. How would you suggest me to resolve this issue?

UPDATE: My site has a large amount of pages (40M) with ~10M indexed. Should I consider adding noindex to some of the pages to make it 'smaller' for search engines?

  • 3
    What is the nature of the site? Do all those 40 million pages have something worthwhile on them? Apr 20, 2013 at 15:29
  • It's an aggregation site. About 10M pages are main pages about entities,and 30M deeper drill-downs per entity.
    – Noam
    Apr 20, 2013 at 15:52

2 Answers 2


Essentially this message means that we (Google) discovered a surprisingly large number of unique URLs while crawling previously-known URLs. This message is sent out before we attempt to crawl those new, unique URLs (since that can take quite some time), it can be useful to inform you of issues with regards to crawlability of your website's structure. Because it's sent out before those new URLs are crawled, the robots.txt, any noindex robots tags, or a rel=canonical are not know at that point.

While it's true that large sites tend to see this message more frequently, it's also the case that especially large sites would profit more by having a clean & crawlable URLs structure from the start. When crawling, we have a limited number of fetches that we can make on a server before it starts to slow down, so if you send us 5-100x more URLs than you actually have content, that can result in us not being able to pick up new content as quickly as we might if we could crawl more efficiently.

My recommendation would be to double-check whether there's something you could do to catch these URLs early (eg avoid linking to them at all, or perhaps use rel=nofollow in the depths of multi-faceted search sections), see if the URL parameter handling tool can be used for your site. Alternately, if you're sure that the search results for your site are "fresh enough," and that the crawling is not a user-noticeable load on your server, then it may be worth just keeping this on your list as something to check when your developers do bigger changes anyway.

  • So, robots.txt and noindex won't actually help? That certainly explains why some sites I've worked on get this message frequently. Some of them have very large numbers of links that are blocked by robots.txt (for link tracking purposes and for constantly changing content). Apr 23, 2013 at 22:19
  • 1
    That's correct, robots.txt & noindex will not prevent this message, but they may each be reasonable means to use in some cases - for example, if the URL is just fetched asynchronously for click-tracking with JavaScript. (using noindex + robots.txt won't work, since we don't see the noindex in those cases.) Apr 24, 2013 at 12:14

There is generally no solution for this "problem". If your site has a large number of pages, you will get this message. When your site has a large number of pages, "an extremely large number of link on your site" is expected.

I get that message on one of my sites that has 10,000 pages. Another site that I worked with that had millions of pages also got that message.

If your site has only a few pages, this could be caused by Googlebot crawling search results or session ids. In such a case, Googlebot will find a large number of urls and links, but you only want a few pages indexed. If that is the case, then you should block Googlebot using robots.txt from search pages or configure your site not to use session ids in the url for googlebot.

There is a discussion about this topic at WebmasterWorld.

  • I have a large number of pages (40M). Should I consider adding noindex to some of them?
    – Noam
    Apr 20, 2013 at 10:40
  • If those pages are all useful content, then you shouldn't noindex any of them. That message in webmaster tools has been there for my sites for years and I've never had problems. Apr 20, 2013 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.