I have a site that Google has indexed 19k pages, in a fairly short amount of time. Bing on the other hand is lagging pretty far behind...51 pages after approximately 3 months of waiting. Many of the items in the sitemap are from search results that are not directly linked from other pages of the site.

From what I've seen, it appears that Bing can index directly linked pages on a site very well, but for other URLs that appear solely in the sitemap, it doesn't even make requests to look at pages, so they have a chance to be indexed.

Has anyone else experienced a problem like this, and what changes can be made to get more pages indexed.

  • @StephenOstermiller Perhaps I should move this answer over to linked duplicate answer and edit to fit as another consideration. – closetnoc Aug 7 '17 at 20:35

The reason for this is simple.

There are two major schools of thought regarding indexing the web.

One is to index everything and then decide what to keep. The advantage of this is that all metrics regarding sites and pages are available to make decisions.

The second is to selectively index pages making any decision before indexing the page. The disadvantage with this is that little can be known about a site or page before indexing. The only thing available are links that exist within the index itself. This is extremely limited.

Google indexes everything and will make decisions later as to whether to keep any page based upon the metrics it can develop.

Bing looks to links on sites within it's own index before indexing other pages.

The reason why this happens is because disk space and systems are expensive. Google has made the decision early on to index everything it can while MSN, now Bing, has decided that indexing everything is too expensive. As well, Google has engineered systems that allow indexing to be both cheaper and faster. For Bing, scaling, while possible, is not within it's business strategy. It is that simple. It is a business decision for Bing not to compete with Google because of scale and expense.

The reason for the difference is also simple. For Microsoft, search is not it's core business, just a part of it. For Google, search IS it's core business and therefore worth investing. Microsoft does not have the will to be all in where for Google, it is a good business strategy and has been the strategy from day one.

If you want more pages indexed by Bing, you simply need to have more links to more pages.

Not the answer you're looking for? Browse other questions tagged or ask your own question.