While checking server logs, I have seen that often the Bing bot requests this URL.
URL: /da-net/products/cadpac/cadpac_jis.shtml
Browser: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Can anyone explain this?
-
Does this URL exist on your website? What is the parent URL?– SteveApr 19, 2011 at 7:06
-
@Steve no this url is not existed, thats why i am curious.– AdeelApr 19, 2011 at 7:07
-
1Are you sure this is the actual Bing spider, and not someone masquerading as them? I would try performing a WHOIS on the IP address attached in your logs; It may be a bot scanning for vulnerabilities to exploit.– Jacob HumeApr 19, 2011 at 18:16
-
How long have you had your domain? It's possible that the page is indexed from a previous owner as it is a very specific URL.– aphexMay 22, 2012 at 8:47
-
@aphex i am managing the domain more than 2 years– AdeelMay 22, 2012 at 9:42
Add a comment
|
1 Answer
Its probably because you have a link to that page. More than likely you have a ../../cadpac/cadpac_jis.shtml link is a url that is on your site.
Bing also reads AJAX urls, so if you have javascript that is creating this url, bing will crawl it.
Easiest way to handle this is just put it in your robots.txt file as a page not to index / carwl.
-
Add a 301 redirect to the correct page using Apache / IIS / php / ASP or whatever you are ussing to run your site. That will fix existing links that are indexed.– FrankNov 13, 2012 at 10:03
-
Bing will also crawl the url if it finds it on other sites. Say if i put it link of your full domain yoursite.com/poop/caca/pickle.html . It will crawl it over and over until you tell it not to.– FrankNov 13, 2012 at 10:08