This question already has an answer here:
When googling my site I get following text under the result for my domain:
"A description for this result is not available because of this site's robots.txt – learn more."
My robots.txt, which resides in the root of my V-host, looks like this:
User-agent: * Disallow:
I did have a problem with this file before since I forgot to change it from the test environment and then I had
User-agent: * Disallow: /
However, I did change it yesterday or the day before.
I have been googling and this seems to be a correct way of handling it as it is now.
Checking the access log of httpd for today, I can see that google has tried to it. But the access_log also shows that Googlebot has been accessing the most of my links as well, feels like ok indexing... Nothing in the error_log from today about robots.txt.
What I cannot find is if google has been indexing /index.php. But I guess folling entry is the same, isn't it?:
What is my problem and how to I resolve it?
When testing in Googles Webmaster Tool I get (this is for my .com domain, all above as well):
Blocked by line 2: Disallow: / Detected as a directory; specific files may have different restrictions
And when doing the exact same thing with my .se domain, which points to the exact same content, I get:
Allowed by line 2: Disallow: Detected as a directory; specific files may have different restrictions