1

I have a lot of problem with Google search, because it (stochastically) finds many duplicates in the form of

http://www.example.com/path/filename.html
http://www.example.com/path//filename.html

I have checked all files throughout and I can't find any error. Does anyone knows any trick how to catch this error or knows what unobvious error could cause it?

I have written the whole site in pure HTML myself.

EDIT Following the suggestion from Stephen Ostermiller, I am sharing my .htaccess

# remove index.html
RewriteCond %{THE_REQUEST} ^.*\/index\.html?
RewriteRule ^(.*)index\.html?$ https://www.example.com/$1 [R=301,L]
# add www.
RewriteCond %{HTTP_HOST} !^www\.
RewriteRule ^(.*)$ https://www.%{HTTP_HOST}/$1 [R=301,L]
# enforce https
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
3
  • Are you running your own web server? What web server are you using? Do you have a site map, and if so, how are you making it? – davidgo Apr 10 at 19:11
  • @davidgo I am simply hosting my web site on cPanel based provider. I am using XML-sitemaps.com generator. I have checked the xml for errors. – Pygmalion Apr 11 at 5:37
  • @davidgo it seems that the provider offers LiteSpeed web server. – Pygmalion Apr 11 at 6:01
2

Double slashes are usually caused by redirect rules. For example RewriteRule ^(.*)$ http://www.example.com/path/$1 [R=301,L] could cause the double slash. The regular expression's parenthesis might be returning /filename.html for the $1 variable. The two slashes come from one from the match and the one hard coded.

To make this redirect work as expected, you could have the rule start with an optional slash that isn't captured: RewriteRule ^/?(.*)$ http://www.example.com/path/$1 [R=301,L] Then the captured group wouldn't contain the leading slash. Alternately, you could remove the second hard coded slash and rely on the captured group having the slash: RewriteRule ^(.*)$ http://www.example.com/path$1 [R=301,L].

I prefer the first of those two solutions. Rewrite rules match leading slashes differently depending on which context they are used in. The URL path does not have a leading slash when rewrite rules are used in .htaccess but the path does have a leading slash when the rewrite rules are used in httpd.conf. Making the leading slash optional with ^/? allows the rule to work correctly in either context.

The problem is compounded by relative links once a crawler finds a URL with a duplicate slash. If http://www.example.com/path//filename.html contains a link like <a href="otherfile.html"> the the crawler will also fetch http://www.example.com/path//otherfile.html. Crawlers will find fewer duplicate slashes on your site if you use "root relative links" on your site. Root relative links start with a slash and link relative to your domain name. If the link were <a href="/path/otherfile.html"> then the crawler wouldn't find additional duplicate slash URLs from one malformed URL.

Once a crawler has found URLs with duplicate slashes, it may continue to crawl them indefinitely, even after you fix all the redirect rules and links. It is a good idea to add rules to remove double slashes from URLs. It looks like this rule set from Issue In Removing Double Or More Slashes From URL By .htaccess is a good one:

# rule 1: remove multiple leading slashes (directly after the TLD)
RewriteCond %{THE_REQUEST} \s/{2,}
RewriteRule (.*) $1 [R=301,L]

# rule 2: remove multiple slashes in the requested path
RewriteCond %{REQUEST_URI} ^(.*)/{2,}(.*)$
RewriteRule (.*) %1/%2 [R=301,L]
7
  • Yes, this is a plausible explanation. I have shared my .htaccess. Would you mind looking at it and see what is wrong? I don't really understand it, I just compose it from examples on the Internet. – Pygmalion Apr 10 at 22:07
  • I'd recommend testing those rules using curl on the command line and see if you get double slashes in the returned Location: field. Something like curl --head https://www.example.com/path/index.html. – Stephen Ostermiller Apr 10 at 22:51
  • Also keep in mind that even if your current rules are fine, you might have had a problem in the past. Bots tend to never forget URLs that give them "200 OK" and continue to crawl them indefinitely, even if the way they got to that URL in the first place has been fixed. – Stephen Ostermiller Apr 10 at 22:53
  • I can't use curl because I only have access to the cpanel – Pygmalion Apr 11 at 5:33
  • curl.se/windows – davidgo Apr 11 at 7:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.