Is it possible to request Google and Bing/Yahoo to drop the cache of links they have indexed?

Question

I radically changed the design of my site to not include embedded session keys which were part of the url itself i.e. /sessionkey/blahblah/blah to just /blahblah/blah.

Now that I have relaunched my site. I am getting hammered with request for URLs which do not exist anymore. I am return 404 Not Found and 410 Gone to the crawlers, but is there anything easier than have them pounding on my server to get a bad response?

I would use their remove feature but all of the session keys indexed were random strings. I would have no idea which session keys to specify. One thing I do know is all of the keys start with a number, I do not know if bots do regex on robots.txt?

Thanks

Answer 1

you have to remember that every page exists forever somewhere, so you shouldn't just remove pages, but provide a 301 redirect (permanent move) to the new pages. That way you're telling Google and other search engines (and provide a transparent transition to visitors) where the pages are now. This is also extremely important for SEO since whatever weight the old pages accumulated in search engines would be lost otherwise.

I'm not familiar with Apache but I'm sure you should be able to add a rule that identifies the incoming request, translate it to the new one, and call a 301 page that points to the correct address.