Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

When the googlebot visits our sites it seemingly tries to visit urls that do not exist resulting in exception being thrown. The url is is like http://ourdomain/thepage.aspx/x/8/x/ etc.x stands for no value in our application. It seems like googlebot seems which parameters it can use, because the urls are no link from the page. Is there any way to control this behaviour?

share|improve this question
3  
Non existent URLs should return a 404 "Not Found" HTTP status code, they should not "throw an exception"? –  w3d Sep 17 '12 at 10:08

1 Answer 1

There is a lengthy article on how to fix crawl errors and on typical problems. Some hints may be of use for you (when to reply with 301 HTTP status codes, when with 404 Status codes etc.).

You should doublecheck if the attempt comes really from the googlebot and is not some attempt from an external source to gain information by playing with the URL. Or, by checking if there is a referer, if it comes from an external link actually.

It's not the same situation here as with infinite loops or "endless" followable links one sometimes sees on result pages with pagination. But you should fix your code to handle the case of "crap" parametrization.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.