Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 0 down vote favorite

This question already has an answer here:

When googling my site I get following text under the result for my domain:

"A description for this result is not available because of this site's robots.txt – learn more."

My robots.txt, which resides in the root of my V-host, looks like this:

User-agent: *
Disallow:

I did have a problem with this file before since I forgot to change it from the test environment and then I had 

User-agent: *
Disallow: /

However, I did change it yesterday or the day before.

I have been googling and this seems to be a correct way of handling it as it is now.

Checking the access log of httpd for today, I can see that google has tried to it. But the access_log also shows that Googlebot has been accessing the most of my links as well, feels like ok indexing... Nothing in the error_log from today about robots.txt.

What I cannot find is if google has been indexing /index.php. But I guess folling entry is the same, isn't it?:

66.249.75.179 - - [07/Feb/2013:15:23:01 +0100] "POST / HTTP/1.1" 200 1 "http://www.mydomain.ext/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

What is my problem and how to I resolve it?

UPDATE

When testing in Googles Webmaster Tool I get (this is for my .com domain, all above as well):

Blocked by line 2: Disallow: / Detected as a directory; specific files may have different restrictions

And when doing the exact same thing with my .se domain, which points to the exact same content, I get:

Allowed by line 2: Disallow: Detected as a directory; specific files may have different restrictions

share|improve this question
Patience... Google's internal processes aren't very fast. Give it a day or two. – Chris S Feb 7 at 18:37
Don't create more than one SE account. Use the same account to update/edit your posts. If you create a question on one account and then abandon that account for another one, how are you going to mark an answer as correct? – Lèse majesté Feb 9 at 15:17

migrated from serverfault.com Feb 7 at 18:36

marked as duplicate by danlefree♦ Feb 7 at 19:32

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1 Answer

up vote 4 down vote

Providing your robots.txt is as you say, like:

User-agent: *
Disallow:

Then based on the fact that you updated it ~12 - 48 hours ago then it could take a while longer before Googlebot reads the updated copy. As far as I'm aware, Google cache the robots.txt file for up to 24 hours (sometimes longer).

If you have a Webmaster Tools account it's probably worth checking the domain for any errors. For more reading check Google's Crawler FAQ

share|improve this answer
Thanks for your answer. I actually update my post with info about the Webmaster Tools info while you posted your answer. I will check out the FAQ. – Nicsoft Feb 7 at 17:48