I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ?

Here is the error message:

webmaster error

Here is my robots.txt file:

User-agent: *
Sitemap: http://mydomain.co.uk/sitemap.xml

Some times that warning means there is a meta noindex tag on your site not just robots.txt blocking the robots. Go to your website and view the source code in a private browser session. Are you using any CMS such as WordPress or another?

Also if you've made changes to allow robots it can take hours or up to 48 hours for Google to recognize the changes and successfully crawl and access your sitemap. You can go to Health on the sidebar in Google webmaster tools and fetch as Googlebot it'll probably give you the same error no matter what page you try and access.

  • ive checked the pages and theres no meta index tags on them, the site went live today (after being hosted with the robots.txt blocking it) so perhaps thats why
    – sam
    Nov 21 '12 at 19:58
  • In that case just hang tight Google takes a few days to recognize a new robots.txt after you were blocking it. I just went through the same thing and Google webmaster tools even fetching the new page is blocked. It'll correct itself in a few days just keep checking it daily by fetching your site as Google bot.
    – Anagio
    Nov 21 '12 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.