1

For last few months, we noticed a large increase in 'Discovered - currently not indexed' pages in Webmaster tools.

By last month, we had almost half of our site pages were classified as discovered but not indexed.

This was quite odd as we had a very well maintained sitemap. We spent months trying to tune and optimise server as Google defines 'Discovered - currently not indexed' as content it want to index but doesn't want to overload the server.

2
  • I've never heard of "Discovered -- currently not indexed" mean that Google would like to index, but doesn't want to overload the server. Do you have a source for that info? Usually it means that the page is low reputation, low quality, or duplicate, ie the reasons laid out in Why aren't search engines indexing my content? Apr 3, 2023 at 0:13
  • We had this problem on a website that turned out to be an issue with a Change of Address. We'd moved from a .com to a .co.uk domain a month earlier, and Google then started to re-index the .com pages (despite the declared change of address and the fact a 301 was in place on the entire .com domain). It took a month to remove the old pages from the index again and get the new pages back up. Apr 3, 2023 at 15:01

1 Answer 1

0

We discovered a very interesting pattern in our logs. While Googlebot had reduced crawling, Bing Bot was crawling crazy amount pages every day.

  • Googlebot 3,000 crawl requests per day and 5,000 per day traffic Bingbot 10k to 45k crawl requests per day for 30-40 per day in traffic

We have added crawl delay directive for Bing Bot since last week and crawl requests have dropped to 2,000 per day. Google bot crawl rate has almost doubled and traffic picking up nicely.

If you are seeing massive increases in discovered but not indexed page, check BingBot crawl activity.

8
  • 2
    Is 40k search crawler requests in a day enough to overload your server? That's a relatively slow consistent rate of one request every 2 seconds, so problems servicing that load could signal an issue with your website's code or hardware provision. The level of overload required to increase your response times enough to back-off Googlebot should not be happening at that request rate even on an intro-level shared plan. If you spent months trying to tune and optimize your server, perhaps it's your host that's the issue. Apr 2, 2023 at 20:03
  • That's not the point. Why does Bing Bot need to make 40k crawl requests each day when it serves 30-40 clicks? We have thousands of archive pages that rarely change. By hitting those pages thousands of times, Bing Bot fills up our cache which can't serve frequently accessed pages then. Better to block Bing Bot altogether as it brings massive downside with almost zero upside.
    – JM John
    Apr 2, 2023 at 22:45
  • The fact that your site only has a few thousand pages is the first hint I've seen that Bingbot is behaving out of the ordinary - I had assumed that it was trying to crawl through tens or hundreds of thousands of pages. I also didn't know that you were working with a limited size rolling cache. All I'm saying is that if my own server were buckling under suspect Bingbot behavior, I would be looking at CPU usage, profiling code, and thinking about what happens if a piece of my site's content generates a few hundred clicks at once. It may not be "the point", but I thought it important to mention. Apr 2, 2023 at 23:42
  • Agree - our site is well set up to cope with serving thousands of hits to a page that is cached. Our hit rate is over 90% except when Bing Bot fills it up in hours and hit rate becomes 20-30% because cache gets filled. I think investing in resources to accomodate poorly written Bingbot makes little sense for traffic it delivers. It is probably collecting data for AI models which benefit us in no way.
    – JM John
    Apr 3, 2023 at 0:01
  • 2
    It is a good ideal to use robots.txt to disallow site search and utm parameters to prevent Google (and Bing) from crawling URLs that shouldn't get crawled or indexed. Apr 3, 2023 at 0:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.