I radically changed the design of my site to not include embedded session keys which were part of the url itself i.e. /sessionkey/blahblah/blah to just /blahblah/blah.
Now that I have relaunched my site. I am getting hammered with request for URLs which do not exist anymore. I am return 404 Not Found and 410 Gone to the crawlers, but is there anything easier than have them pounding on my server to get a bad response?
I would use their remove feature but all of the session keys indexed were random strings. I would have no idea which session keys to specify. One thing I do know is all of the keys start with a number, I do not know if bots do regex on robots.txt?