So I want to set a robots.txt in my site. Pretty straight forward place in root directory and good. But Google spits out a 404 error telling me the robots.txt is not in the place it expects. Not surprising since this happens:

I press view live robots.txt

I then click the "view live robots.txt" button and it takes me to http://www.example.com/www/example.com/robots.txt instead of http://www.example.com/robots.txt.

The question is why is it searching that directory and not just the root? Any help is much appreciated.

  • 3
    It appears that your web server is not configured properly. Unfortunately, there is not enough information here for us to help. Can you post your configuration and any .Htaccess or similar file. Cheers!! – closetnoc May 3 '18 at 1:57
  • My guess is that you redirect to that URL. What is your website so we can check? – Stephen Ostermiller May 3 '18 at 11:57

The problem isn't on Google's end, it is on your end. Your HTTPS is fine:

$ curl --head https://www.qcandles.com/robots.txt
HTTP/1.1 200 OK

However your HTTP to HTTPS redirect is broken for deep links:

$ curl --head http://www.qcandles.com/robots.txt
HTTP/1.1 301 Moved Permanently
Location: https://www.qcandles.com/www/qcandles.com/robots.txt

Somehow you have managed to configure your redirect so that it works fine for the home page:

$ curl --head http://www.qcandles.com/
HTTP/1.1 301 Moved Permanently
Location: https://www.qcandles.com/

To fix this you need to find your rule for redirecting to HTTPS and remove the "/www/qcandles.com" from it. Here is a common mod_rewrite based redirect that you could use instead in your .htaccess file:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{SERVER_NAME}/$1 [R,L]

You could probably also get around this problem by adding your https:// site to Google Search Console and testing your robots.txt file against that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.