Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

While checking server logs, I have seen that often the Bing bot requests this URL.

URL: /da-net/products/cadpac/cadpac_jis.shtml

Browser: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Can anyone explain this?

share|improve this question
    
Does this URL exist on your website? What is the parent URL? –  Steve Apr 19 '11 at 7:06
    
@Steve no this url is not existed, thats why i am curious. –  Adeel Apr 19 '11 at 7:07
1  
Are you sure this is the actual Bing spider, and not someone masquerading as them? I would try performing a WHOIS on the IP address attached in your logs; It may be a bot scanning for vulnerabilities to exploit. –  Jacob Hume Apr 19 '11 at 18:16
    
How long have you had your domain? It's possible that the page is indexed from a previous owner as it is a very specific URL. –  aphex May 22 '12 at 8:47
    
@aphex i am managing the domain more than 2 years –  Adeel May 22 '12 at 9:42

1 Answer 1

Its probably because you have a link to that page. More than likely you have a ../../cadpac/cadpac_jis.shtml link is a url that is on your site.

Bing also reads AJAX urls, so if you have javascript that is creating this url, bing will crawl it.

Easiest way to handle this is just put it in your robots.txt file as a page not to index / carwl.

share|improve this answer
    
Add a 301 redirect to the correct page using Apache / IIS / php / ASP or whatever you are ussing to run your site. That will fix existing links that are indexed. –  Frank Nov 13 '12 at 10:03
    
Bing will also crawl the url if it finds it on other sites. Say if i put it link of your full domain yoursite.com/poop/caca/pickle.html . It will crawl it over and over until you tell it not to. –  Frank Nov 13 '12 at 10:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.