2

While checking server logs, I have seen that often the Bing bot requests this URL.

URL: /da-net/products/cadpac/cadpac_jis.shtml

Browser: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Can anyone explain this?

5
  • Does this URL exist on your website? What is the parent URL?
    – Steve
    Apr 19, 2011 at 7:06
  • @Steve no this url is not existed, thats why i am curious.
    – Adeel
    Apr 19, 2011 at 7:07
  • 1
    Are you sure this is the actual Bing spider, and not someone masquerading as them? I would try performing a WHOIS on the IP address attached in your logs; It may be a bot scanning for vulnerabilities to exploit.
    – Jacob Hume
    Apr 19, 2011 at 18:16
  • How long have you had your domain? It's possible that the page is indexed from a previous owner as it is a very specific URL.
    – aphex
    May 22, 2012 at 8:47
  • @aphex i am managing the domain more than 2 years
    – Adeel
    May 22, 2012 at 9:42

1 Answer 1

3

Its probably because you have a link to that page. More than likely you have a ../../cadpac/cadpac_jis.shtml link is a url that is on your site.

Bing also reads AJAX urls, so if you have javascript that is creating this url, bing will crawl it.

Easiest way to handle this is just put it in your robots.txt file as a page not to index / carwl.

2
  • Add a 301 redirect to the correct page using Apache / IIS / php / ASP or whatever you are ussing to run your site. That will fix existing links that are indexed.
    – Frank
    Nov 13, 2012 at 10:03
  • Bing will also crawl the url if it finds it on other sites. Say if i put it link of your full domain yoursite.com/poop/caca/pickle.html . It will crawl it over and over until you tell it not to.
    – Frank
    Nov 13, 2012 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.