Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website.

The problem is: Whenever I try to crawl the website using any software (DeepCrawl or Screaming Frog), the crawler keeps getting the same page that asks for Language and country for each URL.

Question: How to allow the crawler to bypass, or even select language and country to access the website?

  • 1
    If you are doing an SEO audit and you find that the site is not crawlable because of this cookie, you have already done your job. Googlebot won't be able to crawl the site either. – Stephen Ostermiller Jun 19 '17 at 9:41

Search Engines Bots such as Googlebot do not use cookies and if your content is only visible with cookies then your content is not crawlable. You need to ensure that the website is crawlable without cookies.

The simple solution is to do a cookie check before serving 'choose language option', if the cookie tests negative then you have your website serve the most popular version of your website, then for other languages you use rel="alternate" hreflang="en-XXX" in the head so Google indexes all versions.

  • 1
    Cookie checks often involve redirects that aren't very crawlable either. Ditch the language choice altogether and just have the different languages on different URLs. If you think a user got it wrong, put up a notice in the page giving them an option to click to the correct language. – Stephen Ostermiller Jun 19 '17 at 10:39

You need to use a crawler with a cookie jar. Here's one I wrote some time ago that can log in and keep cookies for a site. You didn't mention a language. This is PHO and Mysql or Oracle.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.