User-agent: * Disallow: /
I don't want search engines to index https://example.com without language prefix. Is that possible?
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.Sign up to join this community
Disallow: / will prevent robots from crawling your entire site. Robots.txt rules are "starts with" rules. Google and other search engines allow some extensions to robots.txt rules such as wild cards and ends with operators. For the robots (including Google) that follow those extensions you could use:
User-agent: * Disallow: /$
Most robots will still crawl your home page because they don't understand the
$ "ends with" operators, but search engine bots should stop crawling just your home page while still crawling the rest of your site.
Another problem is that blocking crawling doesn't always prevent indexing. Google indexes disallowed pages when they have lots of incoming links to them. It won't index, the content of the page, but it show a link to the page in the search results using keywords from the domain name and external link anchor text.
A better option would be to include the tag
<meta name="robots" content="noindex"> in your root page while allowing it to be crawled. That would 100% prevent search engines from including it in their search indexes.
I would suggest that you should have a root page that is crawlable and indexable. It should just allow users to choose their language. When a users language can be identefied from the
Accept-Language header, you could automatically redirect them. For bots that don't send
Accept-Language headers and users that don't have a matching language, you could display a list of languages to choose from. This is what some big sites like Ikea do.
As most answers already said here, I wouldn't recommend doing this, but one way to do would be to set up a 301 redirect to your "main" language (https://example.com/en for example).
Every website needs a default page, so you should choose one and then redirect to it.