6

Since the ForceRecrawl problem went away Bing has come back with some new trends.

I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also looks like I'm not the only one.

I'm not getting these URLs from any other bot, and I regularly run a link checker over my site to check for dead links, so they're not coming from my pages. I wish Bing (and maybe all bots) would at least include one referrer in the request header to let us know where they got the link from (I know they might have more than one reference, but having one is a nice start).

I'm also having trouble understanding Bing's indexing strategy, they index about 25% the number of pages that Google indexes, then they suddenly throw half of them out and start building up again slowly.

Is Bing is trying to alter the URL's and see if it can navigate to pages by "guessing" URL's instead of harvesting them from the normal navigation mechanism? Maybe they cannot master parsing Javascript menu's? I don't know, but they are doing something crazy!

Slightly off topic, but a nice conspiracy theory: There's another bot called "Ezooms/1.0" that's doing something similar: it adds spaces behind dashes it finds in URLs. (I think it's always after the first dash in the URL). By comparing the patterns, I'd almost think these two bots have been written by the same developer (though the mysterious Ezooms has a gmail address added in the user agent string).

Improve this question
4
  • 2
    Are you sure that's actually the bing bot? Check it's ip address.
    – user6901
    May 22, 2012 at 0:01
  • I just checked a few of them, and they are from Microsoft. These for example: 65.52.110.17, 65.52.109.25, 207.46.195.242
    – Louis Somers
    May 22, 2012 at 19:06
  • I'm seeing a similar behavior (Bing adds _2 to some of our URLs). Have you found any solution to this? Thanks, J
    – user27221
    Apr 18, 2013 at 16:02
  • Nope, it did stop after a while though, but they are still crawling content that has been redirected with a 301 permanent redirect to the new location for over 2 years now. I'm wondering how long I'll have to keep those redirects in place (sigh). Googlebot never crawls these any more. I'm still seeing strange BingBot or MSNbot quirks once in a while.
    – Louis Somers
    Apr 18, 2013 at 18:44

1 Answer 1

Reset to default
3

Have you looked at the Bing Webmaster Tools at all?

You can sign up for them and the process of claiming your domains is the same as for Google Webmasters Tools.

These can then give you a full list of crawl stats including links leading to 404s.

Note that if you've removed content you'll see 0 links becuase the not is requesting pages it knew about before.

Another possibilty is that these links are coming from on-page scripts: I had an issue on a site where we were building a link for an advert call in the JavaScript with some it rendered serverside. The bots would find this partial URL in the source and attempt to follow it.

Improve this answer
3
  • Yes sure I have (how would I know how many pages were indexed otherwise?) Any way, under Crawl -> Crawl Details they show a few URLs with 404's, which are correct because they have been removed. However the URL's missing the last letter are not listed there at all, and I'm still getting them (between 3 and 6 of them daily).
    – Louis Somers
    May 25, 2012 at 11:38
  • 1
    How might you know the number of pages indexed? The same way you'd do it on Google: search for site:doodle.co.uk
    – Zhaph - Ben Duguid
    May 25, 2012 at 19:19
  • Cool! Thanks for that tip. I compared what web-master tools indicated with what site:... says. For Google the numbers were equal (some rounding with site:..) but with bing site:... returns 3 times as many results than Bing's web-master tools indicate??? (still less than Google though) This all still does not explain where the urls missing the last letter come from. Am I the only one seeing this?
    – Louis Somers
    May 25, 2012 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.