2

I have just added a new property to the Google search console. I've added it as http://example.com and http://www.example.com.

I receive the following error message: Severe health issues are found in your property. Is robots.txt blocking important pages? Some important page is blocked by robots.txt.

The "some important page" links to the homepage. The error only appears for example.com and not www.example.com. Everything with DNS is resolving and there are no issues. The redirect from www works as well. Google Fetch and Render works as well.

I have removed my robots.txt file, but the issue remains. The robots.txt I was previously using:

User-agent: *
Disallow:

Sitemap: http://example.com/sitemap.xml

I cannot share the domain which I appreciate makes things more difficult to debug. But any reason why I receive an error message when there is no robots.txt file to block Google from crawling?

EDIT:

The crawl errors: "Google couldn't access your site because of a DNS error," Lookup error: "Your DNS server did not recognize your hostname." and Total DNS errors: "Couldn’t communicate with the DNS server." persist. I have used the Fetch as Google tool and it fetches and renders the site with no issues. I have used the robots.txt Tester and there are no issues there either. I have also checked to see if Google has cached the site, which it has. I have been in contact with my DNS provider and they have assured me that there are no issues on there side. I can tell from the Site Errors graph that it was updated yesterday and it shows Errors/Attempts - 0/1. I've never had this happen before on a completely new property. Any suggestions?

Improve this question
2
  • 1
    How long is it since you removed robots.txt? It can take some time before Google will reprocess it.
    – MrWhite
    Feb 26, 2016 at 0:16
  • It has been 20+ hours. No change in the error message. DNS is resolving correctly. robots.txt Tester is showing that there is no file so it will crawl everything: "It seems like you don't have a robots.txt file. In such cases we assume that there are no restrictions and crawl all content on your site."
    – ECsAUtaVku
    Feb 26, 2016 at 12:46

2 Answers 2

Reset to default
1

Google Search Console has a built in tool for testing your robots.txt file: robots.txt tester:

Use that tool to figure out more about what is causing the problem. My guess is that Googlebot isn't seeing the correct version of robots.txt. It may be using an out of date version. DNS may not have propagated yet and it is still seeing the parked domain robots.txt.

It will also allow you to test URLs and highlight any rule that is causing the page to be disallowed.

Improve this answer
4
  • I've used the tool to check every page on the site and everything is being allowed. Google Fetch and Render also works. It's 20+ hours since the error, any suggestions? I've been in contact with my DNS provider and everything is resolving correctly.
    – ECsAUtaVku
    Feb 26, 2016 at 12:49
  • 1
    Wait a week and see if it resolves. Googlebot doesn't work on the timescale of hours.
    – Stephen Ostermiller ♦
    Feb 26, 2016 at 12:57
  • It's been a week and the severe health issue has resolved itself. However several errors persist. I've edited my question to provide more info.
    – ECsAUtaVku
    Mar 3, 2016 at 19:24
  • @ECsAUtaVku Did you ever resolve this?
    – Bill
    Feb 12, 2019 at 18:26
1

There is no need to write url to your sitemap in the robots.txt file, I suggest you remove it from robots.txt and post it on google search console (Webmaster Tools), also this isn't the correct syntax for robots.txt if you are using Disallow you tell bots to not index a webpage, and when leaving it empty this might cause problems in my opinion.

If you feel comfortable with user-agents and crawlers to index every page of your website use User-agent: * Allow: /

And if you want to disallow user-agents and crawlers to index certain pages, do it like that :

User-agent: *
Disallow: /admin/
Disallow: /search/
Disallow: /login/
Disallow: /register/
Improve this answer
2
  • 1
    "when leaving it empty this might cause problems in my opinion." - Disallow: without a path segment is the correct syntax to use in order to allow complete access (although this is the default action so could be omitted entirely). This is according to the original robots.txt "standard". The Allow directive is a modern enhancement which isn't necessarily supported by all bots. Sitemap: is also valid here and is supported by other search engines, not just Google.
    – MrWhite
    Feb 26, 2016 at 0:13
  • About the sitemap I guess you are right, I've never seen it in robots.txt before, no matter how many search engines you are targeting, but you might be right for that, and this part that you mention that allow isn't neccesserily supported by all bots, isn't a disadvantage because all the search engines support allow and this is all that you need for your index, you don't need any spam bots, or at least I don't like them.
    – knif3r
    Feb 26, 2016 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.