I have a search page that uses session to track search criteria made by the user. A session cookie is created automatically when the page is launched. As bots do not use cookie, 1000s of sessions could be created when they crawl, thereby wasting server resources. There are a few solutions which I can think of, but with corresponding side effects:

  1. List this page in robot.txt file, but this would affect SEO for the page.
  2. Check the user agent against a blacklist of known robots before deciding whether to start the session. The list could be very large.
  3. Set a test cookie and do a page redirection to check for the presence of the cookie before starting a session? This means loading the same page twice.
  4. Do away with anonymous sessions and put all criteria in the URL using GET to retrieve them? But this means that I have to verify ALL criteria each time a minor change occur.

What would be a good way to handle this problem?

1 Answer 1


The simple answer here is: Search engine bots do not perform searches, so "1000's of sessions" would not occur.

They won't use sessions either, so you need to provide links in a sitemap that they can crawl containing the search parameters already in them, if you want to index dynamic content that results from searches.

Here's a Google Webmasters Tool answer that should help with how to structure a search URL: URL Structure

Note that you should not add links to a sitemap that might produce duplicate content, even if the search terms in them differ.


Since as further explained, the search page returns default results, you could add rel="nofollow" to the search page so that robots will not follow links on it, but consequently only the default search page results would be indexed, and no further pages.

  • The search page will show the first 10 results based on default criteria if no criteria is specified. Pagination of the results provides links for the crawler to go to the next page of results. Since crawlers don't retain session, a new session would be created each time the crawler move on to the next page of results, ie. /search?page=2 and so on. It seems Google recommends blocking search page from bots. Jun 6, 2013 at 11:14
  • In that case, I would block it. Or you could do number 4 as you wrote, and just suck it up - that's what webmasters sometime have to do :-)
    – dan
    Jun 6, 2013 at 11:22
  • Based on that additional info, I updated my answer for another option.
    – dan
    Jun 6, 2013 at 11:32
  • Thanks for you answer, I think I will just go for option 1 since all the results in the search page is available on my listing page. This is also what stackoverflow did. Jun 14, 2013 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.