3

How can I disallow this URL: http://www.example.com/assets/ico/favicon.png?

Would these lines in my robots.txt work?

User-agent: *
Disallow: /blah/assets*
Disallow: /blah/assets/ico/
1
  • 2
    Why do you have /blah in your robots.txt while the URL starts with /assets (and doesn’t contain "blah")?
    – unor
    Jul 20, 2014 at 16:21

2 Answers 2

6

Both of those would work, but they would also block anything else in those directories. If you only want to block that specific URL, you could do:

Disallow: */blah/assets/ico/favicon.png

Do you have a Google Webmasters Account setup? If so they have a new robots.txt test page where you can see if a URL is blocked or not based on your robots file.

EDIT: My original answer had the below. This would block all URLs that end with favicon.png since it has the $ at the end, but it's pointless in this case since only one favicon.png file can be in that directory. https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961

Disallow: */blah/assets/ico/favicon.png$
5
  • Since when has $ been allowed in robots.txt? Do you have a source for that? Jul 20, 2014 at 12:51
  • support.google.com/webmasters/answer/…. To block any URLs that end in a specific way, use $. I guess it's technically not needed in this case since you can't have more than one favicon.png file in that directory. But it's certainly allowed in robots.txt. Jul 20, 2014 at 14:03
  • @DisgruntledGoat I edited my original answer to say the $ is not needed in this case Jul 20, 2014 at 14:10
  • 1
    Why do you have the * at the beginning of the Disallow path? For parsers following the original robots.txt specification, your Disallow line won’t block the favicon.
    – unor
    Jul 20, 2014 at 16:19
  • @user well technically you could have favicon.png and favicon.pngx if such a file extension existed, so you would need the $ in that case. As noted in my answer below, I would steer clear of using non-standard extensions unless there is no other way to achieve the effect. Jul 20, 2014 at 19:09
2

Firstly, you shouldn't block your favicon image unless you have a very good reason.

What exactly is the 'blah' supposed to represent in your example? If the URL is as you state in the question, you should just use:

Disallow: /assets/ico/favicon.png

That will prevent crawling of the specific image. To block the whole ico directory, use:

Disallow: /assets/ico/

In robots.txt, any path listed matches URLs starting only from the beginning, so the above matches anything beginning with /assets/ico/.

There is no such thing as a * wildcard for Disallow in the robots.txt standard. It is a non-standard extension that according to Wikipedia is only supported by Google.

4
  • 1
    The major search engines (at least Google, Bing, Yahoo, and Ask) support * as a wildcard in disallow in some way. developers.google.com/webmasters/control-crawl-index/docs/… Jul 20, 2014 at 14:36
  • OK, but others such as Duck Duck Go don't appear to (they are at least as popular as Ask). Regardless, the * wildcard isn't necessary in this situation anyway. Jul 20, 2014 at 19:05
  • Depends if this would apply to multiple directories of the same structure, but yes you are probably right. And Duck Duck Go is nowhere near as popular as Ask. Jul 20, 2014 at 22:57
  • @user ok maybe you're right about DDG. I'm constantly hearing about it, but never anything about Ask, so it certainly seems more popular. Jul 21, 2014 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.