3

I set up a website a few weeks ago and I'm trying to get Google to crawl it. When logging into Google's Search Console (Webmaster Tools) and within:

Crawl > Crawl Errors

It reports:

Google couldn't crawl your site because we were unable to access your site's robots.txt file. More Info.

In the "More info" link, Google says I don't need a robots.txt file, so I'm not sure what I need to do to make the site indexed on Google.

Does this affect my site being indexed? How can I fix this issue?

Improve this question
1
  • If your interested in fixing those MIXED CONTENT SSL errors then its because the Google Map V3 MarkerClusterer JS is loading images from http:// rather than https:// download markerclusterer.js locally and amend the HTTP to HTTPS or a better solution would not to use a version that is 2 years or more old. I recommend that you use the latest version hosted on GitHub as the project was abandoned Google Maps Utils Library on Google code awhile back.
    – Simon Hayter
    Mar 9 '16 at 19:16
3

You do not need a robots.txt file for the site to enter Google's index.

Since Google checks every site for a robots.txt your site is returning a 404 error which will return notifications with crawl errors. Simply ignore this error or create an emptyrobots.txt so that your website returns 200 OK status.

It should be noted that a site returning a 404 status is not an ERROR implying that your site requires fixing, for pages that do not exist then a server responding with a 404 status means the server is working as intended.

Improve this answer
4
  • I think there is more to this than simply not having a robots.txt file. You don't normally get the error the OP has posted when you don't have one and it simply returns a 404 - this shouldn't be generating a "crawl error". When you don't have a robots.txt file, Google Search Console simply reports: "It seems like you don't have a robots.txt file. In such cases we assume that there are no restrictions and crawl all content on your site."
    – MrWhite
    Mar 9 '16 at 23:15
  • 1
    @w3dk It does report as a crawl error if you once had one which has been removed and now returns 404s. If you never had one, Google Search Console states what you said. It appears the former is what is happening in this case.
    – grg
    Mar 9 '16 at 23:20
  • 1
    The search console says they got a 500 message for my robots.txt file. So to fix that I put a empty robots.txt file, so the next time they request it, the response would either be a 404 or a 200. Either way would be good enough to get indexed right?
    – Mike
    Mar 10 '16 at 19:59
  • @Mike That bit of information should be added to your question, otherwise everyone reading your question is assuming your server is responding with a 404. If your server responded with a 500 error (which is a generic error - internal to your server) for such a request then that would certainly explain the error reported by Google. Your server should not "normally" be responding with such an error state for a request to robots.txt. This could indicate something more serious, if it persists. Simply creating a robots.txt file would not necessarily resolve this issue.
    – MrWhite
    Mar 11 '16 at 9:47
1

After reading the questions and comments, I would suggest doing any of the following:

  1. Create an robots.txt with only one line in it. Maybe something like this:

    # it works

  2. Or if you don't really want a robots.txt file, then configure your server so that all requests to robots.txt result in an HTTP 410 status code, meaning the file is gone and it should not be requested ever again.

If your server is apache, you can easily add the following contents to .htaccess in the document root folder of your site, or inside the directory tags where the directory is the document root in the main server configuration.

RewriteEngine On
RewriteRule ^robots\.txt$ - [R=410,NC,L]

This will cause any request to robots.txt (regardless of letter casing) to produce an HTTP 410 status code.

I added a forward slash before the dot in the file name to make the dot a literal character instead of a rule-processing character.

The advantage to having a plain robots.txt file as opposed to no robots.txt file is that your error logs won't be filled up with requests to robots.txt.

Improve this answer
1
  • I'm using Firebase Hosting, so no access to my server. I added a blank robots.txt file this moring. I think the original problem was that the robots.txt file was getting a 500 response instead of a 404 or 200 response. Hopefully this fixed it in the next few days.
    – Mike
    Mar 10 '16 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.