Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site.

In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked URLs. It's kind of weird and not logical. Am I missing something?

User-agent : *
Disallow: /*?
Disallow: /wp-login.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Allow: /wp-content/uploads
Disallow: */trackback
Disallow: /*/feed
Disallow: /*/comments
Disallow: /cgi-bin
Disallow: /*.php$
Disallow: /*.inc$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /author/*

I'm afraid my robots.txt doesn't block several URLs I want to block.

Edit (screenshot):
enter image description here

share|improve this question
    
And GWT states "blocked by robots.txt"?! Have you run your robots.txt file through Google's tester, in GWT? You are using a lot of Google-only robots.txt directives - this might not work with other search engines/bots. –  w3d Dec 13 '12 at 19:27
    
In GWT, in menu "Health" => "Blocked URLs", there is a table with four columns named "robots.txt file", "Blocked URLs", "Downloaded" and "Status". First column contains "my-website.com/robots.txt";, second one contains "3", third one contains "Never" and fourth one contains nothing. –  Zistoloen Dec 13 '12 at 20:59
    
It's not mentioned "blocked by robots.txt". I tested robots.txt in Google's tester and it works well. It's just like Google bot found my robots.txt but it mentions that it has never been downloaded!? –  Zistoloen Dec 13 '12 at 21:03
    
Peculiar. That number "3" indicates URLs that have been blocked by robots.txt (hover over the ? next to the column title). How long has your current robots.txt file been 'live'? Is it possible there was a previous robots.txt file that blocked these URLs? –  w3d Dec 13 '12 at 23:37
    
You're right, when I hover ?, I have this text: "Google tried to crawl these URLs in the last 90 days, but was blocked by robots.txt". I don't understand why Google mention my robots.txt has never been downloaded. –  Zistoloen Dec 14 '12 at 9:03

3 Answers 3

up vote 0 down vote accepted

Problem comes from encoding of the robots.txt file. UTF-8 encoding is recommended by Google.

share|improve this answer

It may just be a glitch – they do happen from time to time.

One other possibility occurs though: your screenshot is from GWT on a non-www version of your domain. Do you have the www version authorised as well (e.g., so that you can set preferred domain)? If so, and the www version is canonical, it could explain why Google would know how many URLs are blocked without technically having downloaded the file (they downloaded the www version… which is the same file really, but isn't in a pedantic techie sense).

Bit of a long shot, but that's about the only explanation, other than a glitch, I can come up with.

share|improve this answer
    
Indeed, this is no-www domain. The www version is redirected by 301 to no-www. However, I didn't choose preferred domain in Webmaster Tools because the functionality doesn't work. Moreover, I don't think blocked URLs are www URLs. I blocked by robots.txt many URLs to avoid duplicate content. –  Zistoloen Jan 8 '13 at 11:23
    
Do you actually have GWT verified on both www and non-www though? Incidentally, robots.txt wouldn't normally reference full URLs, so should apply to both www/non-www. –  GDav Jan 8 '13 at 17:02
    
I didn't verify www and no-www. I did it yesterday and I always have 'Never' in Downloaded column. Can I wait a little to see time data in this colum? –  Zistoloen Jan 10 '13 at 15:28

Google will sometimes say it's been blocked by robots.txt when in fact it is a noindex tag preventing Google from indexing the page. If you have any plugins for maintenance mode or others which can possibly set noindex on your home page or other pages I would suggest you go to the page view the source in your browser and check that there is no noindex being rendered by a plugin.

share|improve this answer
    
Thanks Anagio but there is no "noindex" tag on my pages. I edit my question with a screenshot of webmaster tools. –  Zistoloen Jan 3 '13 at 9:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.