1

I just took over maintaining a website and noticed that Google gives "A description for this result is not available because of this site's robots.txt"

So I went to look at the site's robots.txt file and it shows me the following:

User-agent: *
Disallow: /

Which is weird because there wasn't a robots.txt file in the root. So I created one. That file won't display. So I did some research and saw that I should check Settings >> Reading to see if Search Engine Visibility was checked. So I unchecked it (as suggested numerous places). I then went to the site's domain and manually entered the address (www.eonclinics.com/robots.txt) and the bad robots.txt content is still being read, which is weird because I actually have a robots.txt file that looks nothing like the bad one.

I decided to see if it was just me so I renamed the robots.txt file to robot.txt to see if I could see the info inside and sure enough I could. But if I went to "robots.txt" then I would still see the bad content.

So it's as if something is saying "screw you, use this robots.txt file and ignore anything named "robots.txt" in the root."

I thought maybe it was a matter of caching at first, so I cleared my cache and loaded up Google Webmaster tools just in case. Nope - still same bad robots.txt

Yoast is installed and I turned off sitemaps and turned them back on (I saw that could be a trick but it was a 4 year old post) but no avail. There's no robots.txt plugin or google sitemap plugin installed.

Have tried to turn off all plugins but no go.

Hosting is the Deluxe plan from GoDaddy (not my choice) and theme used is "Total" (not my choice). I am stuck with these choices.

If I use http://www.eonclinics.com/robots.txt?nopass=1 then I can see the correct robots.txt file. If I use http://www.eonclinics.com/robots.txt/ then I get the edited robots.txt that is generated in funcitons.php

Has anyone seen a problem like this? Any suggestions? All info I keep coming across is 3 - 5 years old.

Oh - the domain is eonclinics.com.

Improve this question
3
  • For which URL do you get the "A description for this result is not available because of this site's robots.txt" message in SERPs? That site does appear to be indexed OK by Google ("197 results") from what I can see and all results show a valid description - which does not correlate with the "blocking" robots.txt you are reporting?
    – MrWhite
    Commented May 8, 2015 at 9:35
  • I ran the same check as @w3d in Google as well and couldn't see an issue with indexation either.
    – zigojacko
    Commented May 8, 2015 at 13:15
  • Is there a possibility that a different robots configuration is displayed based on country? Run robots.txt through webpagetest.org on multiple systems in different countries and see what happens. If its inconsistent, then maybe someone is messing up your account.
    – Mike -- No longer here
    Commented May 9, 2015 at 15:39

1 Answer 1

Reset to default
2

I think the issue could be locally at your end because I see a robots.txt file at the location that looks like the following:-

sitemap: http://www.eonclinics.com/sitemap.xml

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /archives/
disallow: /*?*
Disallow: *?replytocom
Disallow: /wp-*
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Can you try clearing your browser cache or at least force refreshing (Ctrl + F5 on Windows)?

FWIW, I'd recommend ditching GoDaddy as they screw about with everything in Wordpress with their own turnkey, over-bloated solution that overrides loads of config set in Wordpress. It's horrendous and obtrusive.

Improve this answer
6
  • I also see a different robots.txt than the one in OP, although also different from above (but may have changed in that time)
    – Max
    Commented May 8, 2015 at 9:27
  • @Max Maybe it's a localisation issue? I see the same robots.txt that zigojacko sees. (?) Although I get very different responses for robots.txt?nopass=1 and robots.txt/ as the OP suggests.
    – MrWhite
    Commented May 8, 2015 at 9:38
  • robots.txt?nopass=1 is has the same content as when I view just robots.txt. I'm based in Bangkok
    – Max
    Commented May 8, 2015 at 10:27
  • The scenario from my end (UK) is the same as @w3d describes. It's almost certainly going to my GoDaddy trying to force robots.txt depending on various criteria for their hosted clients.
    – zigojacko
    Commented May 8, 2015 at 13:17
  • To follow up: I found out that the client has everything going through Cloudflare and so there's caching there of the files. When I asked the client about the Cloudflare account and how I can get into it, I was told "I'm paying you to figure these things out. Just fix it" So I guess I need to create a new Cloudflare account, point the DNS away from Cloudflare back to GoDaddy and then re-add the site to Cloudflare and go from there... VERY frustrating. Especially since the email is going through Google and no one knows how to get into the account that's used to setup new accounts there.
    – Chris J. Popp
    Commented May 9, 2015 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.