When googling my site I get following text under the result for my domain:

"A description for this result is not available because of this site's robots.txt – learn more."

My robots.txt, which resides in the root of my V-host, looks like this:

User-agent: *

I did have a problem with this file before since I forgot to change it from the test environment and then I had

User-agent: *
Disallow: /  

However, I did change it yesterday or the day before.

I have been googling and this seems to be a correct way of handling it as it is now.

Checking the access log of httpd for today, I can see that google has tried to it. But the access_log also shows that Googlebot has been accessing the most of my links as well, feels like ok indexing... Nothing in the error_log from today about robots.txt.

What I cannot find is if google has been indexing /index.php. But I guess folling entry is the same, isn't it?: - - [07/Feb/2013:15:23:01 +0100] "POST / HTTP/1.1" 200 1 "http://www.mydomain.ext/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

What is my problem and how to I resolve it?


When testing in Googles Webmaster Tool I get (this is for my .com domain, all above as well):

Blocked by line 2: Disallow: / Detected as a directory; specific files may have different restrictions

And when doing the exact same thing with my .se domain, which points to the exact same content, I get:

Allowed by line 2: Disallow: Detected as a directory; specific files may have different restrictions

  • Patience... Google's internal processes aren't very fast. Give it a day or two. – Chris S Feb 7 '13 at 18:37
  • Don't create more than one SE account. Use the same account to update/edit your posts. If you create a question on one account and then abandon that account for another one, how are you going to mark an answer as correct? – Lèse majesté Feb 9 '13 at 15:17

Providing your robots.txt is as you say, like:

User-agent: *

Then based on the fact that you updated it ~12 - 48 hours ago then it could take a while longer before Googlebot reads the updated copy. As far as I'm aware, Google cache the robots.txt file for up to 24 hours (sometimes longer).

If you have a Webmaster Tools account it's probably worth checking the domain for any errors. For more reading check Google's Crawler FAQ

  • Thanks for your answer. I actually update my post with info about the Webmaster Tools info while you posted your answer. I will check out the FAQ. – Nicsoft Feb 7 '13 at 17:48