3

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment).

An example URL is:

catalog/product-category/product/;jsessionid=Mf87s+Xw2P8ByQYz2CyQjEJh.prod-14?f=p%3A100-200

Can I update my robots.txt to say:

Disallow: /;jsessionid=*

Does anyone see an issue with doing that? I can also canonicalize the pages, but I feel that using robots is a better solution so Googlebot won't have to waste any resources crawling the URLs in the first place.

1
  • Are you linking to the non-jsessionid URLs? What happens when you naviagate to the non-jsessionid URLs?
    – MrWhite
    Oct 27 '15 at 18:50
4

you should exclude such urls from crawling, because you spend your whole crawling budget for those useless urls. Read this topic at Google Product Forum.

set the robots rule like:

disallow: /*jsessionid*
2
  • Should I set the rule like yours, or /*/;jsessionid* ?
    – czmudzin
    Oct 30 '15 at 14:11
  • @czmudzin i would say, use it like in my example
    – Evgeniy
    Oct 30 '15 at 17:03
2
Disallow: /*;jsessionid

A more precise syntax for what you want. A trailing * does nothing.

4
  • Do you have documentation to support this? I know that non-wildcard rules in robots.txt are "starts with" rules and that a wildcard at the end is redundant. But once a rule has a wildcard, I wouldn't necessarily expect it to be interpreted as a "starts with" rule. Sep 28 at 16:34
  • @StephenOstermiller You can check the examples on google's robots.txt page. One of the examples is that /*.php matches /folder/filename.php?parameters. Sep 28 at 16:49
  • 1
    it's remembering me to MrWhite's answer to my question Sep 28 at 21:53
  • Greate answer! "Disallow: /*;jsessionid" includes "Disallow: /*/;jsessionid" and "Disallow: /*jsessionid*". Excellent!
    – skill
    Oct 1 at 23:14
2

Based on official guidelines, don't use the robots.txt file for canonicalization purposes.

You can blocks sessionid URLs using Parameter Handling

If you block using robots.txt, you may lose the search ranking signal from the link to your URL with sessionid/params.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.