0

I am having this issue:

"Google couldn't crawl your site because we were unable to access your site's robots.txt file."

I have already tried all the suggestions provided in the related problems but nothing seems to be working. I am using a plugin that has my site in maintenance mode. Could that be the issue?

I have made a robots.txt file and pushed it in the root directory but the problem remains.

P.S: Some people suggested it was due to the Yoast plugin, so I have deactivated it. Also, I am using Google XML sitemaps and I have unchecked the option where it says to make sure the physical robots.txt is in the root directory. But the problem still remains.

Improve this question
5
  • This is not a wordpress question, and will probably get some down-votes. But you should start here-> google.com/webmasters/tools/… and see what Google says. You should also "fetch as google" from WMT and see the results.
    – Howard E
    Commented May 17, 2016 at 19:27
  • the maintenance plugin might be the issue, have you tried disabling it? How do you check if google is crawling your site? sometimes there are delays before google index it after you submit your site to google
    – Carl Alberto
    Commented May 17, 2016 at 19:28
  • @HowardE I do fetch as google and it successfully fetches my robots.txt, homepage and about, contact us pages. Sometimes it's partial but that's due to the 3rd party stuff.
    – Sana Qureshi
    Commented May 17, 2016 at 19:34
  • @CarlAlberto I have deactivated the plugin, did it 2 days ago. i am checking through GWT. There is always this Yellow Exclamation mark on Robots.txt Fetch. I am totally lost since the Robot.txt tester shows the correct robots.txt file and it's status is 200. Please note that i only have few pages and 2 active posts.
    – Sana Qureshi
    Commented May 17, 2016 at 19:37
  • Your robots.txt file is reachable and looks OK, so providing that your site is not in maintenance mode, you'll need to give Google some time to re-crawl it and update the error status.
    – dan ♦
    Commented May 28, 2016 at 0:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.