1

Example robots.txt:

User-agent: *
Disallow: /

Would it work? I have urls, like https://example.com/en, https://example.com/de

I don't want search engines to index https://example.com without language prefix. Is that possible?

  • Whatever reason you have to do this, it is not good practice. You should have one version of the website as a "default", when it comes to languages, and .com is perfect for that purpose. I would rethink this, if I was you. – Mnea Jan 9 at 12:03
  • You mean, for example: example.com which defaults to english by default? But i have example.com/en for that. – Alexander Kim Jan 9 at 12:05
  • Bear in mind that there are different versions of english as well, so this is not necessarily sufficient. Have a look at hreflang best practices support.google.com/webmasters/answer/189077?hl=en perhaps you will get a better picture. – Mnea Jan 9 at 12:10
  • I already added hreflangs to all of my pages, but google seems to ignore them, when i search my website, i get english and german results together, though my region is set to US. – Alexander Kim Jan 9 at 12:16
  • I'd love to see what it looks like, perhaps I would have better understanding of what you are trying to achieve and why. Perhaps you can send me DM. – Mnea Jan 9 at 14:44
2

Disallow: / will prevent robots from crawling your entire site. Robots.txt rules are "starts with" rules. Google and other search engines allow some extensions to robots.txt rules such as wild cards and ends with operators. For the robots (including Google) that follow those extensions you could use:

User-agent: *
Disallow: /$

Most robots will still crawl your home page because they don't understand the $ "ends with" operators, but search engine bots should stop crawling just your home page while still crawling the rest of your site.

Another problem is that blocking crawling doesn't always prevent indexing. Google indexes disallowed pages when they have lots of incoming links to them. It won't index, the content of the page, but it show a link to the page in the search results using keywords from the domain name and external link anchor text.

A better option would be to include the tag <meta name="robots" content="noindex"> in your root page while allowing it to be crawled. That would 100% prevent search engines from including it in their search indexes.

I would suggest that you should have a root page that is crawlable and indexable. It should just allow users to choose their language. When a users language can be identefied from the Accept-Language header, you could automatically redirect them. For bots that don't send Accept-Language headers and users that don't have a matching language, you could display a list of languages to choose from. This is what some big sites like Ikea do.

  • Thank you, so, basically i just need a page for crawlers/unmatched navigator.language strings to let them choose a language, right? – Alexander Kim Jan 11 at 16:42
  • Yes, that would be a good way of doing it. – Stephen Ostermiller Jan 11 at 16:58
  • But then crawlers would index this page with language choices? – Alexander Kim Jan 12 at 4:07
  • That page isn't going to have many words on it, so it probably isn't going to rank for very much. The only real possibility would be that it ranks for your brand name. You could mitigate that be using hreflang with x-default on that page and point the other languages to the directories. See webmasters.googleblog.com/2013/04/… Even if it did get indexed for your brand name, it wouldn't be horrible user experience. At most users would have to make an extra click. With auto-redirects, most users wouldn't even have to do that. – Stephen Ostermiller Jan 12 at 14:35
0

As most answers already said here, I wouldn't recommend doing this, but one way to do would be to set up a 301 redirect to your "main" language (https://example.com/en for example).

Every website needs a default page, so you should choose one and then redirect to it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.