3

How can I disallow this URL: http://www.example.com/assets/ico/favicon.png?

Would these lines in my robots.txt work?

User-agent: *
Disallow: /blah/assets*
Disallow: /blah/assets/ico/
  • 2
    Why do you have /blah in your robots.txt while the URL starts with /assets (and doesn’t contain "blah")? – unor Jul 20 '14 at 16:21
6

Both of those would work, but they would also block anything else in those directories. If you only want to block that specific URL, you could do:

Disallow: */blah/assets/ico/favicon.png

Do you have a Google Webmasters Account setup? If so they have a new robots.txt test page where you can see if a URL is blocked or not based on your robots file.

EDIT: My original answer had the below. This would block all URLs that end with favicon.png since it has the $ at the end, but it's pointless in this case since only one favicon.png file can be in that directory. https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961

Disallow: */blah/assets/ico/favicon.png$
  • Since when has $ been allowed in robots.txt? Do you have a source for that? – DisgruntledGoat Jul 20 '14 at 12:51
  • support.google.com/webmasters/answer/…. To block any URLs that end in a specific way, use $. I guess it's technically not needed in this case since you can't have more than one favicon.png file in that directory. But it's certainly allowed in robots.txt. – user1795832 Jul 20 '14 at 14:03
  • @DisgruntledGoat I edited my original answer to say the $ is not needed in this case – user1795832 Jul 20 '14 at 14:10
  • 1
    Why do you have the * at the beginning of the Disallow path? For parsers following the original robots.txt specification, your Disallow line won’t block the favicon. – unor Jul 20 '14 at 16:19
  • @user well technically you could have favicon.png and favicon.pngx if such a file extension existed, so you would need the $ in that case. As noted in my answer below, I would steer clear of using non-standard extensions unless there is no other way to achieve the effect. – DisgruntledGoat Jul 20 '14 at 19:09
2

Firstly, you shouldn't block your favicon image unless you have a very good reason.

What exactly is the 'blah' supposed to represent in your example? If the URL is as you state in the question, you should just use:

Disallow: /assets/ico/favicon.png

That will prevent crawling of the specific image. To block the whole ico directory, use:

Disallow: /assets/ico/

In robots.txt, any path listed matches URLs starting only from the beginning, so the above matches anything beginning with /assets/ico/.

There is no such thing as a * wildcard for Disallow in the robots.txt standard. It is a non-standard extension that according to Wikipedia is only supported by Google.

  • 1
    The major search engines (at least Google, Bing, Yahoo, and Ask) support * as a wildcard in disallow in some way. developers.google.com/webmasters/control-crawl-index/docs/… – user1795832 Jul 20 '14 at 14:36
  • OK, but others such as Duck Duck Go don't appear to (they are at least as popular as Ask). Regardless, the * wildcard isn't necessary in this situation anyway. – DisgruntledGoat Jul 20 '14 at 19:05
  • Depends if this would apply to multiple directories of the same structure, but yes you are probably right. And Duck Duck Go is nowhere near as popular as Ask. – user1795832 Jul 20 '14 at 22:57
  • @user ok maybe you're right about DDG. I'm constantly hearing about it, but never anything about Ask, so it certainly seems more popular. – DisgruntledGoat Jul 21 '14 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.