While checking server logs, I have seen that often the Bing bot requests this URL.
URL: /da-net/products/cadpac/cadpac_jis.shtml
Browser: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Can anyone explain this?
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this communityWhile checking server logs, I have seen that often the Bing bot requests this URL.
URL: /da-net/products/cadpac/cadpac_jis.shtml
Browser: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Can anyone explain this?
Its probably because you have a link to that page. More than likely you have a ../../cadpac/cadpac_jis.shtml link is a url that is on your site.
Bing also reads AJAX urls, so if you have javascript that is creating this url, bing will crawl it.
Easiest way to handle this is just put it in your robots.txt file as a page not to index / carwl.