I have a robots text file which needs to mass exclude certain urls, currently setup as so
Disallow: /somestring Disallow: /*/somestring
However, some of my URLs generated by users are coming back like:
I would like to
Allow: this type of URL. In other words, if there are no other characters after a pattern match like
/*/somestring exclude, if there are other characters, include.
Is there a way around this, or a robots directive appropriate in this index?