How to configure the robots.txt file with a multilingual site?

Question

I have a site with Drupal 8 and I activate English and French (Language from the URL path prefix). Here is what my url resembles :

https://www.example.com/en/node/add

https://www.example.com/fr/node/add

And here is the current configuration of my robots.txt file :

# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/

How to configure the robots.txt file with a multilingual site ?

Do I have to add fr and praise each url or will it do it alone ?

Answer 1

Your question is not clear. Are you trying to block any version of your site?

Robots.txt is used to disallow or allow search engines from crawling particular locations of your website.

Also, your current config is wrong, use the following:

# Paths (clean URLs)
User-agent: *
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/

Sitemap: https://www.example.com/sitemap.xml

Also, you do not need to list out every single URL from a sub-directory. For example, if you want to block the SE to not crawl/index "user", you can use the below code:

# Paths (clean URLs)
User-agent: *
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/

Sitemap: https://www.example.com/sitemap.xml

You can know more about how to use robots.txt using Search Console resources.

Answer 2

Disallow values need to start with the beginning of the URL path.

So Disallow: /node/add blocks https://www.example.com/node/add, but neither https://www.example.com/en/node/add nor https://www.example.com/fr/node/add.

If you want to block all variants of the node add page, you have to use:

Disallow: /node/add
Disallow: /en/node/add
Disallow: /fr/node/add

(Note that I omitted the trailing slash. If the node add page doesn’t have the trailing slash, the Disallow value must omit it, too, otherwise it wouldn’t match. If the node add page does have the trailing slash, omitting it in the Disallow value is no problem, it will still match.)