0

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment).

An example URL is:

catalog/product-category/product/;jsessionid=Mf87s+Xw2P8ByQYz2CyQjEJh.prod-14?f=p%3A100-200

Can I update my robots.txt to say:

Disallow: /;jsessionid=*

Does anyone see an issue with doing that? I can also canonicalize the pages, but I feel that using robots is a better solution so Googlebot won't have to waste any resources crawling the URLs in the first place.

1
  • Are you linking to the non-jsessionid URLs? What happens when you naviagate to the non-jsessionid URLs? – MrWhite Oct 27 '15 at 18:50
1

you should exclude such urls from crawling, because you spend your whole crawling budget for those useless urls. Read this topic at Google Product Forum.

set the robots rule like:

disallow: /*jsessionid*
2
  • Should I set the rule like yours, or /*/;jsessionid* ? – czmudzin Oct 30 '15 at 14:11
  • @czmudzin i would say, use it like in my example – Evgeniy Oct 30 '15 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.