1

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ?

Here is the error message:

webmaster error

Here is my robots.txt file:

User-agent: *
Disallow:
Sitemap: http://mydomain.co.uk/sitemap.xml
Improve this question

1 Answer 1

Reset to default
1

Some times that warning means there is a meta noindex tag on your site not just robots.txt blocking the robots. Go to your website and view the source code in a private browser session. Are you using any CMS such as WordPress or another?

Also if you've made changes to allow robots it can take hours or up to 48 hours for Google to recognize the changes and successfully crawl and access your sitemap. You can go to Health on the sidebar in Google webmaster tools and fetch as Googlebot it'll probably give you the same error no matter what page you try and access.

Improve this answer
2
  • ive checked the pages and theres no meta index tags on them, the site went live today (after being hosted with the robots.txt blocking it) so perhaps thats why
    – sam
    Commented Nov 21, 2012 at 19:58
  • In that case just hang tight Google takes a few days to recognize a new robots.txt after you were blocking it. I just went through the same thing and Google webmaster tools even fetching the new page is blocked. It'll correct itself in a few days just keep checking it daily by fetching your site as Google bot.
    – Anagio
    Commented Nov 21, 2012 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.