Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools.

This is what I have in robots.txt:

User-agent: *
Allow: /*?p=
Allow: /index.php/blog/
Allow: /catalog/seo_sitemap/category/
Allow: /catalogsearch/result/
Allow: /media/
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Disallow: /contents/
Disallow: /contents/fr/
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /admin/
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID= 
Disallow: /*?limit=all

I copied this from Magento's default robots.txt and I added some folders that don't need to be indexed.

share|improve this question

closed as unclear what you're asking by John Conde Apr 27 at 9:01

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question.If this question can be reworded to fit the rules in the help center, please edit the question.

    
First thing, I can't believe that you have cron ah publicly available. Or in fact any config/installation scripts. Your opening yourself to getting hacked –  Liam Sorsby Apr 27 at 7:59
    
@LiamSorsby - It's Magento, it's special... Everything is loaded in the htdocs folder system. And protected by .htaccess files. Make your judgement on running Magento for your ecommerce website from that or any other Zend framework application for that matter. –  Fiasco Labs Apr 27 at 17:37
    
Just because it is magento, doesn't meant it is special it's a well known fact if you leave an executable file on the server that is publicly accessible, someone will find a glitch and hack it. –  Liam Sorsby Apr 27 at 19:26
    
@LiamSorsby - preaching to the choir on this one... You should have inhabited the MagentoCommerce forums in the day when they had traffic. Every so often some web store owner would post in breathless amazement on how they'd cured a problem by deleting one of the .htaccess files. Guess what that enables? No Sarcasm markup for the it's special dig on stackexchange. ;^) –  Fiasco Labs Apr 28 at 3:25
    
Not sure if it's just me but that just doesn't make any sense –  Liam Sorsby Apr 28 at 5:47

1 Answer 1

up vote 2 down vote accepted

Google gives you that message when it finds out that many pages on your site that it had been indexing are now all blocked in robots.txt. Whether or not that message is legitimate for you depends on whether you actually want those blocked pages crawled.

I'm not specifically familiar with Magento, but the following blocked URLs seem like they might contain important content that you want indexed in search engines:

Disallow: /contents/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /newsletter/

In addition to checking that those aren't disallowed by mistake you should:

  • Run a site:example.com search on Google and see what they have indexed. Check to make sure there isn't anything important that is now blocked by robots.txt
  • Examine the URLs in your sitemap. Nothing in your sitemap should be blocked by robots.txt
  • Use Google Webmaster Tool's "Fetch as Google" tool to make sure that they can actually download some URLs that shouldn't be blocked.

I would advise against disallowing the crawling of all JavaScript and CSS files. You should remove the Disallow: /*.js$ and Disallow: /*.css$. Google has said that they really want to be able to render your pages. They do so for screenshot as well and ranking purposes. Your site may not rank as well if they can't do this.

share|improve this answer
    
Thank you Stephen very much your answer was helpful –  Khalil Apr 27 at 20:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.