There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site.

In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked URLs. It's kind of weird and not logical. Am I missing something?

User-agent : *
Disallow: /*?
Disallow: /wp-login.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Allow: /wp-content/uploads
Disallow: */trackback
Disallow: /*/feed
Disallow: /*/comments
Disallow: /cgi-bin
Disallow: /*.php$
Disallow: /*.inc$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /author/*

I'm afraid my robots.txt doesn't block several URLs I want to block.

Edit (screenshot):
enter image description here

  • And GWT states "blocked by robots.txt"?! Have you run your robots.txt file through Google's tester, in GWT? You are using a lot of Google-only robots.txt directives - this might not work with other search engines/bots.
    – MrWhite
    Dec 13, 2012 at 19:27
  • In GWT, in menu "Health" => "Blocked URLs", there is a table with four columns named "robots.txt file", "Blocked URLs", "Downloaded" and "Status". First column contains "my-website.com/robots.txt", second one contains "3", third one contains "Never" and fourth one contains nothing.
    – Zistoloen
    Dec 13, 2012 at 20:59
  • It's not mentioned "blocked by robots.txt". I tested robots.txt in Google's tester and it works well. It's just like Google bot found my robots.txt but it mentions that it has never been downloaded!?
    – Zistoloen
    Dec 13, 2012 at 21:03
  • Peculiar. That number "3" indicates URLs that have been blocked by robots.txt (hover over the ? next to the column title). How long has your current robots.txt file been 'live'? Is it possible there was a previous robots.txt file that blocked these URLs?
    – MrWhite
    Dec 13, 2012 at 23:37
  • You're right, when I hover ?, I have this text: "Google tried to crawl these URLs in the last 90 days, but was blocked by robots.txt". I don't understand why Google mention my robots.txt has never been downloaded.
    – Zistoloen
    Dec 14, 2012 at 9:03

3 Answers 3


It may just be a glitch – they do happen from time to time.

One other possibility occurs though: your screenshot is from GWT on a non-www version of your domain. Do you have the www version authorised as well (e.g., so that you can set preferred domain)? If so, and the www version is canonical, it could explain why Google would know how many URLs are blocked without technically having downloaded the file (they downloaded the www version… which is the same file really, but isn't in a pedantic techie sense).

Bit of a long shot, but that's about the only explanation, other than a glitch, I can come up with.

  • Indeed, this is no-www domain. The www version is redirected by 301 to no-www. However, I didn't choose preferred domain in Webmaster Tools because the functionality doesn't work. Moreover, I don't think blocked URLs are www URLs. I blocked by robots.txt many URLs to avoid duplicate content.
    – Zistoloen
    Jan 8, 2013 at 11:23
  • Do you actually have GWT verified on both www and non-www though? Incidentally, robots.txt wouldn't normally reference full URLs, so should apply to both www/non-www.
    – GDVS
    Jan 8, 2013 at 17:02
  • I didn't verify www and no-www. I did it yesterday and I always have 'Never' in Downloaded column. Can I wait a little to see time data in this colum?
    – Zistoloen
    Jan 10, 2013 at 15:28

Google will sometimes say it's been blocked by robots.txt when in fact it is a noindex tag preventing Google from indexing the page. If you have any plugins for maintenance mode or others which can possibly set noindex on your home page or other pages I would suggest you go to the page view the source in your browser and check that there is no noindex being rendered by a plugin.

  • Thanks Anagio but there is no "noindex" tag on my pages. I edit my question with a screenshot of webmaster tools.
    – Zistoloen
    Jan 3, 2013 at 9:07

Problem comes from encoding of the robots.txt file. UTF-8 encoding is recommended by Google.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.