I crawled my Wordpress site using the Screaming Frog program and noticed a few issues which might have SEO implications and was hoping someone could help identify the problem (if there is one).

N.B: I have obscured the domain name in the image as I do not want it made public.

The canonical URL for my domain is prefixed by https://www but as you can see from the tree graph image below, there are a number of URLs accessible under the http protocol and the non-www version of the domain. Those URLs are non-indexable but I'm curious as to why they're accessible at all. I was thinking that shouldn't happen if the proper redirects were in place.

In the Wordpress admin settings I have entered the correct version of my domain for the Wordpress and site addresses (i.e., https://www)

It's possible my .htaccess file might require editing so I have pasted it below:

## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
<IfModule LiteSpeed>
RewriteEngine on
CacheLookup on
RewriteRule .* - [E=Cache-Control:no-autoflush]
RewriteRule \.object-cache\.ini - [F,L]

### marker CACHE RESOURCE start ###
RewriteRule wp-content/.*/[^/]*(responsive|css|js|dynamic|loader|fonts)\.php - [E=cache-control:max-age=3600]
### marker CACHE RESOURCE end ###

### marker FAVICON start ###
RewriteRule favicon\.ico$ - [E=cache-control:max-age=86400]
### marker FAVICON end ###

### marker DROPQS start ###
CacheKeyModify -qs:fbclid
CacheKeyModify -qs:gclid
CacheKeyModify -qs:utm*
CacheKeyModify -qs:_ga
### marker DROPQS end ###

## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
### marker BROWSER CACHE start ###
<IfModule mod_expires.c>
ExpiresActive on
ExpiresByType application/pdf A31557600
ExpiresByType image/x-icon A31557600
ExpiresByType image/vnd.microsoft.icon A31557600
ExpiresByType image/svg+xml A31557600

ExpiresByType image/jpg A31557600
ExpiresByType image/jpeg A31557600
ExpiresByType image/png A31557600
ExpiresByType image/gif A31557600
ExpiresByType image/webp A31557600

ExpiresByType video/ogg A31557600
ExpiresByType audio/ogg A31557600
ExpiresByType video/mp4 A31557600
ExpiresByType video/webm A31557600

ExpiresByType text/css A31557600
ExpiresByType text/javascript A31557600
ExpiresByType application/javascript A31557600
ExpiresByType application/x-javascript A31557600

ExpiresByType application/x-font-ttf A31557600
ExpiresByType application/x-font-woff A31557600
ExpiresByType application/font-woff A31557600
ExpiresByType application/font-woff2 A31557600
ExpiresByType application/vnd.ms-fontobject A31557600
ExpiresByType font/ttf A31557600
ExpiresByType font/otf A31557600
ExpiresByType font/woff A31557600
ExpiresByType font/woff2 A31557600

### marker BROWSER CACHE end ###

## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
#This Apache config file was created by Duplicator Installer on 2021-02-17 10:08:29.
#The original can be found in archived file with the name .htaccess__[HASH]

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

# END WordPress

enter image description here

1 Answer 1


The best way to not make the site accessible is not to serve it. You can block bots by denying access via the user-agent string but that can be spoofed by malicious crawlers.

The example provided makes not attempts at redirecting traffic through the desired URL which lets users and bots crawl the site via 4 versions (https://www.example.com, http://www.example.com, https://example.com and http://example.com). By adding a few lines to your .htaccess file, you can permanently redirect traffic to the three undesired protocol and host combinations which means you will no longer be serving pages with URLs you don't want. There are different ways to do this but here is one:

RewriteEngine on

RewriteCond %{HTTPS} off
RewriteRule ^/?(.*)$ https://www.example.com/$1 [R=301,L]

RewriteCond %{HTTPS} on
RewriteCond %{HTTP_HOST} !^www.example.com$
RewriteRule ^/?(.*)$ https://www.example.com/$1 [R=301,L]

Without the above or something equivalent, all 4 versions are accessible and even without links to the URLs you don't want, I found that they eventually get accessed occasionally. It's not clear why but I've seen it many times in logs.

Since this is a new site, you may even not redirect at all and return 404 not found since there is no incoming links to preserve but it's usually good to redirect in case users type the site from memory and don't enter it the way you expect.

  • Thanks @Itai. I thought the redirections were handled automatically by Wordpress based on the URLs entered in the admin settings. The site's not new, by the way. It's about 5 years old.
    – jrcollins
    Commented Mar 5, 2021 at 7:33
  • 1
    OK. No, you to redirect yourself to controls the URLs to access you site. WP does some redirection for post-names, numbers, categories but the rest is up to you. Since you mentioned you didn't want to make the site public, I assumed it was new. In any case, I recommend redirecting rather than returning 404 at least for people to find it even if they don't remember if it's www.example.com or example.com
    – Itai
    Commented Mar 5, 2021 at 14:09
  • 1
    You should not be able to read any http page if there is a redirect. So there must be pages that are cached. Specially with CND, caches can be spread across many servers and it might take a while for them to be flushed or expire. Presumably your analysis is not using the local cache either, but if it is, its a good idea to clear it too.
    – Itai
    Commented Mar 6, 2021 at 0:54
  • 1
    There is no impact to SEO for anything that is not indexable. Even if it was, using canonical meta tag, potential issues can be neutralized. They are quite rate too because the http/https and www/non-www version is very common and easily recognizable by the search engines (Google even has setting you can set the preference). Canonical mostly helps when you have more complex URLs that show the same results in different orders, languages, etc or sometime with different branding.
    – Itai
    Commented Mar 6, 2021 at 19:24
  • 1
    A heuristic will guess but that could mean an inconsistent use of http and https in search results.
    – Itai
    Commented Mar 7, 2021 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.