I have submitted a sitemap which has many thousands of URLs, but when I look at the webmaster tools it claims that 9800 of my URLS are blocked by my robots.txt file.

What am I supposed to do to convince it that nothing is being blocked?

Shows my site is basically being blocked by robots.txt

Shows my robots.txt is blank

  • The URLS in question are they new or old?... – Simon Hayter Sep 25 '14 at 22:55
  • They are old. They existed before and now seemed to be blocked by a robots.txt which is blank. – weexpectedTHIS Sep 25 '14 at 23:32
  • Webmaster Tools is sometimes wrong and shouldn't always be trusted. If your robots is blank then theres nothing to block those URLS. It's worth mentioning that on your screenshot it says! Showing data from the last year So display only 30 days worth. Also do a site: example.com in Google to see if any urls have dropped from the index which I very much doubt. – Simon Hayter Sep 26 '14 at 6:06

First of all check your robots.txt and add below mentioned commands

User-agent: *

Sitemap : www.yourdoamin.com/sitemap.xml

After adding this resubmit the robots file to website as well as google webmaster tool.

Try this and let me know if you have any query regarding this. For more details visit : http://www.robotstxt.org/robotstxt.html

| improve this answer | |
  • This doesnt fix the problem, his robots.txt is empty, it can't block anything. This only indicates where to find tje sitemap, which is submitted via the webmatertools – Martijn Sep 26 '14 at 7:08
  • Ya i know that but it will help the bots to index the pages of the website. sometimes they got confused. – PushP Sep 26 '14 at 7:14
  • Sometimes robots got confused thus the above addition will help them to crawl the pages of the website. Since it is not wise to kept robots file empty. – PushP Sep 26 '14 at 7:17
  • But the pages are found (so the sitemap works), they don't get added because somethings blocks them. – Martijn Sep 26 '14 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.