I have a website with the following domain name:

https://example.com (there is no index.html in the root domain name)

I have the following redirect based on language detection in .htaccess:

https://example.com > https://example.com/en/index.html (for English language) https://example.com > https://example.com/fr/index.html (for French language)

The user is redirected correctly, but Bing site scan returns error 403 due to the missing index.html in the root domain https://example.com. This means that Bing's crawler fails indexing the website. Google's crawler is not affected and can index the website properly.

What are the best SEO friendly solutions to this problem I am experiencing with Bing?

That is my current .htaccess redirect/rewrite

RewriteCond     %{SERVER_PORT} ^80$
RewriteRule     ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [L,R=301]

# Detect browser's language
RewriteCond %{HTTP:Accept-Language} ^en [NC]
RewriteRule ^$ https://example.com/en/ [L,R=301] 
RewriteCond %{HTTP:Accept-Language} ^fr [NC]
RewriteRule ^$ https://example.com/fr/ [L,R=301]
  • What is the language detection code that you are using in .htaccess? Can you edit your question to include it? May 26, 2023 at 11:43
  • Hi, I post the code in the Question for clarity. Here it would be hard to read. BTW: did you receive my reply in private with all references about EU GDPR? May 26, 2023 at 11:48
  • It is always preferable to edit your questions and answers here rather than providing updates in the comments. Your reply was received. May 26, 2023 at 12:14
  • Do you have any suggestion/recommendation regarding the issue detailed in the question? May 26, 2023 at 12:17
  • 1
    How is Googlebot finding the fr site? When it crawls, it sends Accept-language: en. I'd expect Googlebot to be ignoringc your fr site entirely unless you have links to it from somewhere else. May 26, 2023 at 12:34

1 Answer 1

# Detect browser's language
RewriteCond %{HTTP:Accept-Language} ^en [NC]
RewriteRule ^$ https://example.com/en/ [L,R=301] 
RewriteCond %{HTTP:Accept-Language} ^fr [NC]
RewriteRule ^$ https://example.com/fr/ [L,R=301]

The user is redirected correctly

The user is only redirected "correctly" if en or fr is the user's primary language (or rather, if the language code appears at the start of the Accept-Language header). This is unreliable (even when it "works") for a number reasons:

  • The primary language is not guaranteed to appear at the start of the Accept-Language header.
  • The Accept-Langauge header can contain a series of weighted languages - which is not easily determined by a simple regex.
  • If the user's browser is set to another "primary" language and en or fr is only a secondary language then the redirect will fail.
  • The user is using someone else's machine where the primary language is different to what they understand.

It's possible that Bing is not sending an Accept-Language header at all (or en or fr is not at the start of the Accept-Language header) so the redirect is failing and a 403 Forbidden response is served (since there is no index document). You would need to set a "default" language.

For example, if en is to be the default:

# Detect browser's language
RewriteCond %{HTTP:Accept-Language} ^(en|fr) [NC]
RewriteRule ^$ https://example.com/%1/ [L,R=302]

# Default to "en"
RewriteRule ^$ https://example.com/en/ [L,R=302] 

This at least avoids the 403.

The %1 backreference contains the language code as captured in the preceding condition, ie. the value at the start of the Accept-Language header - if en or fr.

Also note the 302, as opposed to 301. Language redirects should not be 301 (permanent) - it should be 302 (temporary). By making it a 301 then the user is unable to change the language since the 301 redirect is cached persistently by the browser.

However, other search engines (such as Googlebot) generally only send an Accept-Language: en header so they are always redirected to the English version. Although, you are only redirecting the homepage (root directory), so providing the fr language version is referenced in other ways (Sitemap, hreflang, etc) then it should still be crawlable.

However, there is potentially another issue....

(From your cross-site question on StackOverflow)
There is a structure like the following: example.com/en , example.com/en/sub1, example.com/en sub2, example.com/en/etc

If a request is received for /sub1 or /sub2 (no language code) then should this be redirected or result in a 404 (or 403)? (Currently the later, since the above redirect will not match, which is perhaps desirable. Requests for /sub1 or /sub2 without a language code might be rare anyway.)

Users should be given the option of changing the language (since the Accept-Language header is not reliable - as discussed above). So, if the user selects another language (which is perhaps then stored in a cookie by your script) then this can be used to override the auto-redirection.

For example, if the cookie used is called lang with a value of either en or fr (if the user has chosen this) then you could add the following before the above language redirects:

# Redirect user if previously selected a language (overriding default)
RewriteCond %{HTTP_COOKIE} (?:^|;|\s)lang=(en|fr)(?:$|;)
RewriteRule ^$ https://example.com/%1/ [L,R=302]

# Language redirect follows...

The %1 backreference contains the language code as captured in the preceding condition, ie. the value of the lang cookie (if any).

  • 1
    I'm not sure how the subdirectories figuer into this. ^$ in the rewrite rules should match exactly the home page URL, not /sub1. May 26, 2023 at 14:04
  • @StephenOstermiller Yes, and maybe that's desirable / for the better.
    – MrWhite
    May 26, 2023 at 14:49
  • 1
    It is never a good idea to auto redirect from one language to another, so making sure that auto-redirects don't include /fr or /en better for sure. May 26, 2023 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.