I have a search page that uses session to track search criteria made by the user. A session cookie is created automatically when the page is launched. As bots do not use cookie, 1000s of sessions could be created when they crawl, thereby wasting server resources. There are a few solutions which I can think of, but with corresponding side effects:
- List this page in robot.txt file, but this would affect SEO for the page.
- Check the user agent against a blacklist of known robots before deciding whether to start the session. The list could be very large.
- Set a test cookie and do a page redirection to check for the presence of the cookie before starting a session? This means loading the same page twice.
- Do away with anonymous sessions and put all criteria in the URL using GET to retrieve them? But this means that I have to verify ALL criteria each time a minor change occur.
What would be a good way to handle this problem?