38

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs.

So far I have the following files:

  • /favicon.ico
  • /robots.txt
  • /sitemap.xml
  • /apple-touch-icon-72x72-precomposed.png
  • /apple-touch-icon-72x72.png
  • /apple-touch-icon-precomposed.png
  • /apple-touch-icon.png

Are there other files that get requested often and should be on every website?

Improve this question
8
  • 2
    I just hate those! WTF are they thinking? If I wanted them to use it I would have added them...
    – PeeHaa
    Dec 15, 2011 at 16:33
  • 1
    I understand you... But what I hate even more are 404 messages in my apache log.
    – Raffael Luthiger
    Dec 15, 2011 at 17:16
  • 2
    to be clear. What I hate is that user-agents tries to access them by default.
    – PeeHaa
    Dec 15, 2011 at 17:18
  • This begs a different question though. It seems like your primary concern is to reduce "extraneous" 404 logs. Are you getting a lot of legitimate 404 logs that you need to sort through? Or are you trying to sort through the log for other errors and the 404's are getting in the way?
    – CLo
    Dec 15, 2011 at 18:10
  • 1
    I don't have a problem with browsers looking for site icons automatically, as that saves webmasters the need to include a reference to them on every single page. However, I do wish Apple would have stuck with the favicon standard, which supports alpha channels as well as multiple icon sizes.
    – Lèse majesté
    Feb 15, 2012 at 1:06

5 Answers 5

Reset to default
21

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder.

Here are all the ones I know of:

  • favicon.ico
    Gives your pages an icon in tabs, bookmarks, etc.
  • robots.txt
    Useful if you need to block any pages from search engines, but can be blank to allow access to everything. It can also contain the location for your sitemap.
  • Webmaster Tools verification files
    The easiest method of verification (saves having useless meta tags in your HTML code). There's google[hash].html and BingSiteAuth.xml. I have one for Yahoo in my root too but that's probably no longer needed.
  • crossdomain.xml
    This file is requested by Adobe Flash any time Flash needs to connect to your site. You will probably never see this file in your logs on a small site, but I had a bunch of requests for it on one site. You can just create a blank file since the default is to block access to external sites.
  • .htaccess (Apache)
    Obviously not requested by browsers/bots but vital for most sites.

Finally, regarding the icons that the iPhone insists on requesting: there are several sizes that may get requested. My iPhone 3G (iOS 5) requests the 57x57 version and I see 72x72 and 114x114 (most common) in my error logs. So if you want to eliminate all 404s from your logs, mind-boggingly you would need to provide all of these:

  • /apple-touch-icon-57x57-precomposed.png
  • /apple-touch-icon-57x57.png
  • /apple-touch-icon-72x72-precomposed.png
  • /apple-touch-icon-72x72.png
  • /apple-touch-icon-114x114-precomposed.png
  • /apple-touch-icon-114x114.png
  • /apple-touch-icon-precomposed.png
  • /apple-touch-icon.png
Improve this answer
5
  • 1
    .htaccess is specific to Apache and, even for those using Apache, it is a poor choice for performance if one has access to the VirtualHost definitions.
    – danlefree
    Dec 16, 2011 at 0:42
  • +1 for crossdomain.xml - that hadn't occurred to me, and is a great idea to include.
    – Jacob Hume
    Dec 16, 2011 at 2:36
  • I had once a bot requesting the sitemap.xml about once a day. And I never registered this file somewhere. So this bot was just guessing it.
    – Raffael Luthiger
    Dec 16, 2011 at 19:25
  • @DisgruntledGoat: Thank you for the update. I didn't know that those other sizes exists too.
    – Raffael Luthiger
    Dec 19, 2011 at 20:58
  • 1
    Silverlight also uses clientaccesspolicy.xml.
    – Lèse majesté
    Feb 15, 2012 at 1:08
3

I think you probably have it.

Even apple touch icon is not entirely nescessary, since if you choose not to have it (and don't link to it) then iOS will just use a thumbnail of the page (personally, I add them in though).

I'd say you only really NEED favicon.ico and robots.txt, sitemap.xml is optional too.

Improve this answer
1
  • 1
    This wouldn't prevent the request, and thus the 404 errors that Raffael is seeing in their logs. If you were going to concentrate on which files make the most impact, I'd say you're right.
    – Jacob Hume
    Dec 16, 2011 at 2:34
2

Another one that I've noticed is browserconfig.xml which is some bastardisation requested by IE. Bloody rubbish filling up the logs.

Improve this answer
1
  • Thanks. Do you know what should be in this file?
    – Raffael Luthiger
    Jan 8, 2014 at 15:47
1

Newer apple versions request these files (in this order)

  • /apple-touch-icon-120x120-precomposed.png
  • /apple-touch-icon-120x120.png
  • /apple-touch-icon-precomposed.png
  • /apple-touch-icon.png
Improve this answer
1
  • Thanks. This would extend the list from DisgruntledGoat.
    – Raffael Luthiger
    Oct 9, 2014 at 10:18
0

Instead of providing these files why not 410 them?

Apache config:

Redirect    gone    "/favicon.ico"
Redirect    gone    "/sitemap.xml"
Redirect    gone    "/crossdomain.xml"
Redirect    gone    "/apple-touch-icon"
Redirect    gone    "/clientaccesspolicy.xml"
Redirect    gone    "/browserconfig.xml"
Improve this answer
2
  • because we want to make as many devices as possible happy. The question is about giving the users the most comfort by providing the browsers whatever they need.
    – Raffael Luthiger
    Aug 27, 2020 at 19:52
  • Many of them can be provided at alternate locations with link tag in the head. Especially the favicon.ico. Which is BTW on the way out being replaced by *.png. This is 2020. The some of the answers here are stale.
    – NOYB
    Aug 27, 2020 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.