While checking server logs, I have seen that often the Bing bot requests this URL.
URL: /da-net/products/cadpac/cadpac_jis.shtml
Browser: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Can anyone explain this?
2
-
Does this URL exist on your website? What is the parent URL? – Steve Apr 19 '11 at 7:06
-
@Steve no this url is not existed, thats why i am curious. – Adeel Apr 19 '11 at 7:07
-
1Are you sure this is the actual Bing spider, and not someone masquerading as them? I would try performing a WHOIS on the IP address attached in your logs; It may be a bot scanning for vulnerabilities to exploit. – Jacob Hume Apr 19 '11 at 18:16
-
How long have you had your domain? It's possible that the page is indexed from a previous owner as it is a very specific URL. – aphex May 22 '12 at 8:47
-
@aphex i am managing the domain more than 2 years – Adeel May 22 '12 at 9:42
3
Its probably because you have a link to that page. More than likely you have a ../../cadpac/cadpac_jis.shtml link is a url that is on your site.
Bing also reads AJAX urls, so if you have javascript that is creating this url, bing will crawl it.
Easiest way to handle this is just put it in your robots.txt file as a page not to index / carwl.
-
Add a 301 redirect to the correct page using Apache / IIS / php / ASP or whatever you are ussing to run your site. That will fix existing links that are indexed. – Frank Nov 13 '12 at 10:03
-
Bing will also crawl the url if it finds it on other sites. Say if i put it link of your full domain yoursite.com/poop/caca/pickle.html . It will crawl it over and over until you tell it not to. – Frank Nov 13 '12 at 10:08
