3

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment).

An example URL is:

catalog/product-category/product/;jsessionid=Mf87s+Xw2P8ByQYz2CyQjEJh.prod-14?f=p%3A100-200

Can I update my robots.txt to say:

Disallow: /;jsessionid=*

Does anyone see an issue with doing that? I can also canonicalize the pages, but I feel that using robots is a better solution so Googlebot won't have to waste any resources crawling the URLs in the first place.

Improve this question
1
  • Are you linking to the non-jsessionid URLs? What happens when you naviagate to the non-jsessionid URLs?
    – MrWhite
    Oct 27, 2015 at 18:50

3 Answers 3

Reset to default
4

you should exclude such urls from crawling, because you spend your whole crawling budget for those useless urls. Read this topic at Google Product Forum.

set the robots rule like:

disallow: /*jsessionid*
Improve this answer
2
  • Should I set the rule like yours, or /*/;jsessionid* ?
    – czmudzin
    Oct 30, 2015 at 14:11
  • @czmudzin i would say, use it like in my example
    – Evgeniy
    Oct 30, 2015 at 17:03
2 
Disallow: /*;jsessionid

A more precise syntax for what you want. A trailing * does nothing.

Improve this answer
4
  • Do you have documentation to support this? I know that non-wildcard rules in robots.txt are "starts with" rules and that a wildcard at the end is redundant. But once a rule has a wildcard, I wouldn't necessarily expect it to be interpreted as a "starts with" rule.
    – Stephen Ostermiller ♦
    Sep 28, 2021 at 16:34
  • @StephenOstermiller You can check the examples on google's robots.txt page. One of the examples is that /*.php matches /folder/filename.php?parameters.
    – Maximillian Laumeister ♦
    Sep 28, 2021 at 16:49
  • 1
    it's remembering me to MrWhite's answer to my question
    – Muhammad Dyas Yaskur
    Sep 28, 2021 at 21:53
  • Greate answer! "Disallow: /*;jsessionid" includes "Disallow: /*/;jsessionid" and "Disallow: /*jsessionid*". Excellent!
    – Skill
    Oct 1, 2021 at 23:14
2

Based on official guidelines, don't use the robots.txt file for canonicalization purposes.

You can blocks sessionid URLs using Parameter Handling

If you block using robots.txt, you may lose the search ranking signal from the link to your URL with sessionid/params.

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.