6

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as

# Files
User-agent: * 
Disallow: /mage

But this also block URLs start with mage like magenta-color-item.html.

How I write in robot to block mage only not URL start with mage?

Improve this question

1 Answer 1

Reset to default
9

You can add a dollar sign to the end of the string which means it will only match exactly that entry:

# Files
User-agent: * 
Disallow: /mage$

This will only block the mage file if it come straight after the root domain:

www.example.com/mage

If there are any other preceding directories, you must add these o the entry. So to block the file located below:

www.example.com/somedirectory/mage

You would need to use:

# Files
    User-agent: * 
    Disallow: /somedirectory/mage$
Improve this answer
5
  • 3
    It's worth adding that this is an extension to the robots exclusion protocol and my not be understood by all crawlers, though it is by those used for google, bing and yahoo which are the three of most value today. It can mean that some other crawlers would hit /mage because it doesn't begin with /mage$ and they don't understand the $ extension. If it's truly vital that /mage never be crawled then other mechanisms are needed (e.g. never having something else begin with /mage, or restricting access to that resource so bots just can't see it).
    – Jon Hanna
    Apr 28, 2014 at 9:24
  • 3
    Another possibility is to disallow /mage and allow /magenta-color-item.html etc. While Allow: is also an extension, this approach is more conservative; those crawlers that understand that extension (which again includes the bigger search engines' crawlers, and Allow is probably more widely implemented than $) will hit /magenta-color-item.html and while those that don't know that extension won't, at least they are more reliably going to avoid /mage. This might be preferable.
    – Jon Hanna
    Apr 28, 2014 at 9:27
  • thanks @JonHanna Your second comment suggest to disallow /mage and allow /magenta-color-item.html but in my case there are lots of URLs like /magenta-color-item.html with variation. In fact Google webmaster show almost 250 different URLs blocked, which are start with mage and all these URLs are generated Automatically by Magento using logic based on user data. How to handle this scenario then?
    – Sandesh
    Apr 28, 2014 at 11:35
  • I'd suggest just following Max's answer in that case. It's a good answer (I +1'd it anyway). I think the stuff I mentioned in the comments are worth considering sometimes too, but Max's remains good for many people (why I didn't add a different answer) and by the sounds of it, definitely the way to go for yourself.
    – Jon Hanna
    Apr 28, 2014 at 11:55
  • Wouldn't it be easier and more proper to block mage via .htaccess rules? It is an executable that shouldn't be accessed through the web browser anyway, as in the command line only for module downloads and installs. With the robots.txt thing, you're only telling proper search engines to ignore it, it's still accessible for everyone else.
    – Fiasco Labs
    Oct 21, 2014 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.