3

When the googlebot visits our sites it seemingly tries to visit urls that do not exist resulting in exception being thrown. The url is is like http://ourdomain/thepage.aspx/x/8/x/ etc.x stands for no value in our application. It seems like googlebot seems which parameters it can use, because the urls are no link from the page. Is there any way to control this behaviour?

1
  • 3
    Non existent URLs should return a 404 "Not Found" HTTP status code, they should not "throw an exception"?
    – MrWhite
    Commented Sep 17, 2012 at 10:08

1 Answer 1

1

There is a lengthy article on how to fix crawl errors and on typical problems. Some hints may be of use for you (when to reply with 301 HTTP status codes, when with 404 Status codes etc.).

You should doublecheck if the attempt comes really from the googlebot and is not some attempt from an external source to gain information by playing with the URL. Or, by checking if there is a referer, if it comes from an external link actually.

It's not the same situation here as with infinite loops or "endless" followable links one sometimes sees on result pages with pagination. But you should fix your code to handle the case of "crap" parametrization.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.