4

I have a weird problem where for some reason search engines see that I have duplicate URLs, like https://hi@example.com/x and https://example.com/x

I'd like the first (i.e. with hi@) to 301 redirect to the second URL (i.e. without hi@), via .htaccess.

I've tried code like the below but it doesn't work (the other redirect rules I have in that .htaccess file do work though). Any ideas?

<IfModule mod_rewrite.c>
<IfModule mod_negotiation.c>
    Options -MultiViews
</IfModule>

RewriteEngine On

RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

# Redirect Trailing Slashes If Not A Folder...
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)/$ /$1 [L,R=301]    

RewriteCond %{HTTP_Host} hi@example\.com$ [NC]
RewriteRule ^(.*)$ https://example.com/$1 [L,R=301]

</IfModule>
  • 1
    hi@ isn't part of the host. hi is the name of the logged in user. – Stephen Ostermiller Mar 23 '18 at 10:24
  • If that's the case, it shouldn't get picked up by search engines, since search engines wouldn't be able to log in / see logged-in versions of pages. – user6122500 Mar 23 '18 at 10:25
  • It is a basic auth/digest auth user name. Most sites don't use that style of login at all. It's the type where the browser pops up a username/password box in a modal dialog. Not the type where you put the username and password into a form on the site. – Stephen Ostermiller Mar 23 '18 at 10:29
3

As @Stephen suggested in comments, hi is the username part when the username:password is specified as part of the URL (See RFC 1738) in Basic HTTP Authentication.

If that's the case, it shouldn't get picked up by search engines

That depends if other sites have (maliciously) linked to the URL in this format.

hi@ is not actually sent as part of the Host: HTTP request header, so you can't match it with the HTTP_HOST server variable in Apache .htaccess. Compliant user-agents will strip this from the URL and create a base64 encoded Authorization: header with this information before making the request.

If you are not using HTTP Authentication then you could potentially redirect/block any request that contains something in the Authorization HTTP request header. For example, you can try the following at the top of your .htaccess file (not at the end):

RewriteCond %{HTTP:Authorization} !^$
RewriteRule (.*) https://example.com/$1 [R=301,L]

Although this is a little risky, as it is dependent on the user-agent not sending the Authorization header again in the second request in order to avoid a redirect loop. Although this shouldn't be a problem for "real users" in this instance, as they shouldn't be "logged in" anyway. Googlebot should be shown the URL less the login credentials (although will possibly experience a redirect loop and not actually see your real content - but will avoid the duplicate content issue). If you were genuinely using HTTP Authentication on your site then this would certainly result in a redirect loop if the user had "logged in".

Alternatively, serve a 410 Gone instead for such requests (I wouldn't think you're getting much SEO value from these URLs anyway):

RewriteCond %{HTTP:Authorization} !^$
RewriteRule ^ - [G]

UPDATE: As @StephenOstermiller suggested in comments, the technically correct response is probably to return a 401 Unauthorized instead. This has the added benefit of logging the user out (if they really were logged in), so the Authorization header would not be sent in subsequent requests. And Google should not return any 4xx response in the SERPs. For example, similar to above:

RewriteCond %{HTTP:Authorization} !^$
RewriteRule ^ - [R=401]

It's worth noting that it's unlikely that any of this will affect real users following such links, as it seems that modern browsers tend to strip this information entirely before making the request (for reasons of security). It is removed from the visible URL and no Authorization header is actually sent. This appears to have been removed from IE a long time ago apparently. And also seems to be the case in my tests on Chrome 64 (and Opera). The following Chrome bug report backs this up and suggests this feature has been removed - the bug report is closed with a status of "WontFix".

The following ServerFault question has much discussion in comments about whether this URL format "works" (strangely, some users state that this does work in Chrome - although these comments are possibly outdated):

UPDATE: Note that RFC 3986 (which updates RFC 1738 - linked above) formerly deprecates the use of "username:password" in the URL

Use of the format "user:password" in the userinfo field is deprecated.

  • 1
    I think 401 Unauthorized would the correct response compared to 410 Gone. That should cause browsers to "log the user out" based on StackOverflow's How to log out user from web site using BASIC authentication? – Stephen Ostermiller Mar 23 '18 at 12:17
  • Thanks! The RewriteCond %{HTTP:Authorization} !^$ RewriteRule (.*) example.com/$1 [R=301,L] seems to have generated the outcome I wanted (301 for any hi@... URLs) – user6122500 Mar 23 '18 at 22:36
  • @StephenOstermiller Ah yes, you have a point. (If the browser was actually "logged in" then I don't think the redirect would work, since I would expect the Authorization header to get sent back in the redirected request?) – MrWhite Mar 24 '18 at 3:02
  • Even if the Authorization is sent again, the 301 redirect uses an absolute URL and should remove the user name from the URL. That should solve the problem for search engines. If that header is unused, it should be harmless as long as it is not in the URL. I just wouldn't recommend 410 Gone. – Stephen Ostermiller Mar 24 '18 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.