0

I have a site with Drupal 8 and I activate English and French (Language from the URL path prefix). Here is what my url resembles :

https://www.example.com/en/node/add

https://www.example.com/fr/node/add

And here is the current configuration of my robots.txt file :

# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/

How to configure the robots.txt file with a multilingual site ?

Do I have to add fr and praise each url or will it do it alone ?

0

Disallow values need to start with the beginning of the URL path.

So Disallow: /node/add blocks https://www.example.com/node/add, but neither https://www.example.com/en/node/add nor https://www.example.com/fr/node/add.

If you want to block all variants of the node add page, you have to use:

Disallow: /node/add
Disallow: /en/node/add
Disallow: /fr/node/add

(Note that I omitted the trailing slash. If the node add page doesn’t have the trailing slash, the Disallow value must omit it, too, otherwise it wouldn’t match. If the node add page does have the trailing slash, omitting it in the Disallow value is no problem, it will still match.)

3

Your question is not clear. Are you trying to block any version of your site?

Robots.txt is used to disallow or allow search engines from crawling particular locations of your website.

Also, your current config is wrong, use the following:

# Paths (clean URLs)
User-agent: *
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/

Sitemap: https://www.example.com/sitemap.xml

Also, you do not need to list out every single URL from a sub-directory. For example, if you want to block the SE to not crawl/index "user", you can use the below code:

# Paths (clean URLs)
User-agent: *
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/

Sitemap: https://www.example.com/sitemap.xml

You can know more about how to use robots.txt using Search Console resources.

  • What I mean is that now in my url there is "fr" and "en" for the language of the site. Should I include this in my robots.txt file ? – ML19 Feb 6 at 10:04
  • If you are trying to block any URLs under "fr" and "en" versions of the site, then you can update it in the robots.txt file. Example: You have example.com/user and example.com/fr/user, then you should use Disallow: /fr/user/ to block the whole sub-directory – idk Feb 7 at 5:43

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.