4

My site's URL is: www.hackinguniverse.org

From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt".

My robots.txt is:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.hackinguniverse.org/feeds/posts/default?orderby=updated

Now for other information - I hosted this website on blogger. I have some other sitemaps too where I had only included "/Search" in Disallow like in this robots.txt file. But those sites are ok. I mean no message is showing on those. So why am I getting that message telling that I have blocked some important page via robot.txt?

2
  • 3
    "Allow" directives are generally redundant, unless they are replacing previous directives. They don't help and they can make the code confusing. The purpose of robots.txt is to exclude url's. By design everything that is not excluded can be crawled.
    – Osvaldo
    Commented Jan 20, 2012 at 23:19
  • @Osvaldo Yah... But can I change robot.txt for Google Bloger.. Because i am not able to change it..
    – Rounak
    Commented Jan 21, 2012 at 4:08

1 Answer 1

4

Check the download date on the robots file, it could be referring to an old version of the file.

1
  • Oh... I dont know but Google Automaticly fixes the problem !!
    – Rounak
    Commented Jan 21, 2012 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.