2

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title.

The Webmaster Tool reports this error:

Crawl allowed?

 No: blocked by robots.txt Page fetch  Failed: Blocked by robots.txt

Here is my robots.txt which is pretty simple and allow all requests:

User-agent: *
Allow: /

I confirmed I don't have any HTML tag blocking the index, my meta robots is <meta name="robots" content="index, follow" />

Not sure why I'm getting this error message. The robots.txt tester (https://www.google.com/webmasters/tools/robots-testing-tool) reports no error, but yet I'm getting this problem for > 3 weeks now.

Also the root url return HTTP 200

HTTP/1.1 200 OK
Date: Sat, 15 Sep 2018 18:41:34 GMT
Content-Type: text/html
Connection: keep-alive
Last-Modified: Sat, 15 Sep 2018 18:20:13 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 45ad3a6cab3192d6-SJC

Any idea what could be the problem? This is only happening for my root URL.

Improve this question
3
  • 1
    Hard to say what it is, without knowing the site URL especially, but I would update your syntax from Allow: / to Disallow: (i.e. Disallow nothing) and see if that changes anything the next time your site is crawled. Also, make sure there isn't a conflicting robots.txt anywhere on your site that may have been generated by some plugin or other.
    – Henry Visotski
    Commented Sep 17, 2018 at 20:29
  • 1
    Does Google Search Console say when the last crawl date was? Maybe it hasn't been crawled since last time you updated your robots.txt file.
    – Stephen Ostermiller ♦
    Commented Sep 17, 2018 at 20:57
  • I haven't changed the robots and actually tried to completely removed Disallow and only keep Allow, same problem. As for the 2nd question, Google tried to fetch my root url every day (Last Crawl Sep 17, 2018, 4:20:04 PM).
    – woop
    Commented Sep 18, 2018 at 1:22

1 Answer 1

Reset to default
1

This could be due to following reasons:

  1. Caching (Your website or DNS caching your website and Google is unable to read your updated robots.txt file.)

  2. Change syntax of your robots.txt file to:

    User-agent: *
Disallow:
Improve this answer
3
  • Already changed the robots.txt a couple of days ago to completely remove Disallow, same problem. As for the caching I haven't change anything (my robots is the same for multiple months)
    – woop
    Commented Sep 18, 2018 at 1:23
  • Are you using CloudFlare or any caching plugin ?
    – Nandla
    Commented Sep 18, 2018 at 17:15
  • Yep I'm using CloudFlare. Haven't changed anything though. I verified I don't have any WAF / IP Blacklist / User Agent Blacklist and force cache purge, still no luck. Also, I tried to use the 'old' Google Webmaster consol and "Fetch and Render" as google and it worked
    – woop
    Commented Sep 18, 2018 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.