1

I first started seeing this a little over two months ago on a site I run but now I am seeing it on several others I manage. It appears the bot is trying to parse the URL into folders and this is what's causing the issue.

For instance this URL is correct:

http://amgoa.org/Proposed-Alaska-Gun-Law-SCR6/State-Law/8895

But then Bing tries to access this URL:

http://amgoa.org/Proposed-Alaska-Gun-Law-SCR6/State-Law

And then this URL:

/Proposed-Alaska-Gun-Law-SCR6 (sorry cant post more than 2 complete links)

The second two of course throwing a 404.

This site has over 67,000 pages and this error on their part is driving us nuts, loading up the error logs with tens of thousands of 404 for URLs that are incorrect.

About a month ago I built an xml sitemap script hoping that would solve the issue. I submitted it to Google and Bing via the webmaster tools section. Google correctly indexed all 67,000+ pages while Bing is sitting there trying to parse these non existent URLs.

Has anyone else seen this and more importantly does anyone know how to either stop this or contact Bing to get them to stop?

Improve this question

1 Answer 1

Reset to default
2

The problem you have isn't Bing but the way your server is handling error responses.

Your 404 pages are reporting: SERVER RESPONSE: HTTP/1.1 200 OK

It should be reporting: SERVER RESPONSE: HTTP/1.0 404 NOT FOUND

So search engines are assuming they are valid pages and that's why they are being crawled all the time. Fix this and Bing should start to stop smashing at those pages.

You can test your header response using FireBug, Google Webmaster Tools or these online websites: site-scan.com, seobook.

The odd thing is that you do have status 404 Not Found in your header response but its not valid since 200 OK on the first response, basically you have a soft 404 which Yahoo and Bing don't take seriously

Improve this answer
8
  • I am sending this: header("HTTP/1.0 404 Not Found"); From my custom page not found page
    – user973767
    Jun 8, 2013 at 20:18
  • Yep I can confirm your site is now doing that, but before it wasn't.
    – Simon Hayter
    Jun 8, 2013 at 22:33
  • Thank you, the PHP code I had before was copied and pasted from another site and didn't include the HTTP part. Hopefully this will stop this nightmare and thanks for pointing it out to me
    – user973767
    Jun 9, 2013 at 1:42
  • Well so much for that idea, lasted one day and now they are back at it again ... same exact nonsense
    – user973767
    Jun 12, 2013 at 14:22
  • It can take weeks to months for search engines to remove those pages, even unindexed pages will get crawled, Don't alarm yourself to much with the crawling process of bing or Google the issue here is that they are indexed. Google for instance will crawl that page 2-3 times before it actually removes it, its kinda like a grace period because if it removed it quickly webmasters could lose their rankings by mistake.
    – Simon Hayter
    Jun 12, 2013 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.