Google sitemap access error, something to do with .htaccess redirect from http://www. to http://

Question

I recently updated the .htaccess of my site to redirect from http://www. to http://.

It all works fine but now I've made some changes to my site and uploaded a new sitemap.xml to Google Search Console (formerly Google Webmaster Tools). I'm getting the following message:

"We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
General HTTP error: Domain name not found"

I made the sitemap using xml-sitemaps.com, any idea how I might resolve this?

---

The domain is: http://example.com

The sitemap is: http://example.com/sitemap.xml

The .htaccess is:

Options +FollowSymlinks
RewriteEngine on
rewritecond %{http_host} ^www.example.com [nc]
rewriterule ^(.*)$ http://example.com$1 [r=301,nc] 

Answer 1

One thing you should check is your preferred domain in Google Webmaster Tools. If you've told Google to prefer www.lightandspace.co.uk over just lightandspace.co.uk, but you now have the former redirecting to the latter, that might well cause some crawling issues.

More generally, it seems that the "General HTTP error: Domain name not found" error is typically caused by DNS issues. However, both of your hostnames resolve just fine for me, even via Google's public DNS servers, so if that's the problem then I'm unable to reproduce it.

Answer 2

(Old question, but...)

rewriterule ^(.*)$ http://lightandspace.co.uk$1 [r=301,nc]

This directive is invalid in a .htaccess (or directory) context and will certainly result in a DNS error if requesting the www subdomain. (This affects all URLs, except the document root, not just the /sitemap.xml URL. So I'm not sure what "It all works fine" is referring to specifically?)

Specifically, the substitution is missing a slash after the hostname. The $1 backreference does not contain a slash prefix (as would seem to be expected) because in a .htaccess context the URL-path that the RewriteRule pattern matches against does not contain a slash prefix. (The URL-path that is matched against is less the directory-prefix. The directory-prefix always ends with a slash, so the URL-path that is matched never starts with a slash. This behaviour is different to when the directive is used in a server or virtualhost context - in this case the RewriteRule pattern matches against the full URL-path, which naturally does start with a slash.)

For example, if Google is requesting http://www.example.com/sitemap.xml then your site will redirect this to http://example.comsitemap.xml, which will simply fail to resolve and a "General HTTP error: Domain name not found" error will be reported.

So, in a .htaccess context, you need to explicitly include the slash. For example:

RewriteRule (.*) http://example.com/$1 [R=301,L]

Note also, the NC flag was superfluous here (you are matching everything, regardless of case). And you probably need the L flag to terminate further processing of your directives (if you have more directives). Regex is greedy by default, so ^(.*)$ is the same as (.*) - it matches everything.

Alternatively, you can use the REQUEST_URI server variable, which contains the full URL-path, including the slash prefix:

RewriteRule .* http://example.com%{REQUEST_URI} [R=301,L]