This is my robots.txt:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz
But Google Webmaster Tools tells me that robots is blocking access to the sitemap:
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit: URL restricted by robots.txt.
I read that Google Webmaster Tools caches robots.txt, but the file has been updated more than 36 hours ago.
Hitting TEST sitemap does not cause Google to fetch a new sitemap. Only SUBMIT sitemap was able to do that. (BTW, I dont see whats the point in 'test sitemap' unless you paste your current sitemap in there - it doesnt fetch a fresh copy of the sitemap from the address it asks you to enter before the test - but thats a question for another day.)
After submitting (instead of testing) a new sitemap the situation changed. I now get "URL blocked by robots.txt. Sitemap contains URLs which are blocked by robots.txt." for 44 URLs. There are exactly 44 URLs in the sitemap. This means that Google is using the new sitemap but it is still going by the old robots rule (which kept everything off-limits) None of the 44 URLs are in
/wp-includes/ (which is kind of impossible anyways, since robots.txt is built on the fly by the same plugin that creates the sitemap).
It gets worse: on a Google Search results page, the description for the homepage reads: "A description for this result is not available because of this site's robots.txt – learn more". All other pages have fine descriptions. There is no robots.txt OR robots meta blocking indexing of the homepage.