My site's URL is: www.hackinguniverse.org

From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt".

My robots.txt is:

User-agent: Mediapartners-Google

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.hackinguniverse.org/feeds/posts/default?orderby=updated

Now for other information - I hosted this website on blogger. I have some other sitemaps too where I had only included "/Search" in Disallow like in this robots.txt file. But those sites are ok. I mean no message is showing on those. So why am I getting that message telling that I have blocked some important page via robot.txt?

  • 3
    "Allow" directives are generally redundant, unless they are replacing previous directives. They don't help and they can make the code confusing. The purpose of robots.txt is to exclude url's. By design everything that is not excluded can be crawled. – Osvaldo Jan 20 '12 at 23:19
  • @Osvaldo Yah... But can I change robot.txt for Google Bloger.. Because i am not able to change it.. – Rounak Jan 21 '12 at 4:08

Check the download date on the robots file, it could be referring to an old version of the file.

| improve this answer | |
  • Oh... I dont know but Google Automaticly fixes the problem !! – Rounak Jan 21 '12 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.