Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 0 down vote favorite

im getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt but its not, has any one come across this or a way to resolve this before ?

the error message

webmaster error

here is whats in my robots.txt 

User-agent: *
Disallow:
Sitemap: http://mydomain.co.uk/sitemap.xml
share|improve this question

1 Answer

up vote 1 down vote accepted

Some times that warning means there is a meta noindex tag on your site not just robots.txt blocking the robots. Go to your website and view the source code in a private browser session. Are you using any CMS such as WordPress or another?

Also if you've made changes to allow robots it can take hours or up to 48 hours for Google to recognize the changes and successfully crawl and access your sitemap. You can go to Health on the sidebar in Google webmaster tools and fetch as Googlebot it'll probably give you the same error no matter what page you try and access.

share|improve this answer
ive checked the pages and theres no meta index tags on them, the site went live today (after being hosted with the robots.txt blocking it) so perhaps thats why – sam Nov 21 '12 at 19:58
In that case just hang tight Google takes a few days to recognize a new robots.txt after you were blocking it. I just went through the same thing and Google webmaster tools even fetching the new page is blocked. It'll correct itself in a few days just keep checking it daily by fetching your site as Google bot. – Anagio Nov 21 '12 at 20:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.