I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs.

So far I have the following files:

  • /favicon.ico
  • /robots.txt
  • /sitemap.xml
  • /apple-touch-icon-72x72-precomposed.png
  • /apple-touch-icon-72x72.png
  • /apple-touch-icon-precomposed.png
  • /apple-touch-icon.png

Are there other files that get requested often and should be on every website?

  • 2
    I just hate those! WTF are they thinking? If I wanted them to use it I would have added them... – PeeHaa Dec 15 '11 at 16:33
  • 1
    I understand you... But what I hate even more are 404 messages in my apache log. – Raffael Luthiger Dec 15 '11 at 17:16
  • 2
    to be clear. What I hate is that user-agents tries to access them by default. – PeeHaa Dec 15 '11 at 17:18
  • This begs a different question though. It seems like your primary concern is to reduce "extraneous" 404 logs. Are you getting a lot of legitimate 404 logs that you need to sort through? Or are you trying to sort through the log for other errors and the 404's are getting in the way? – Chris Dec 15 '11 at 18:10
  • 1
    I don't have a problem with browsers looking for site icons automatically, as that saves webmasters the need to include a reference to them on every single page. However, I do wish Apple would have stuck with the favicon standard, which supports alpha channels as well as multiple icon sizes. – Lèse majesté Feb 15 '12 at 1:06

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder.

Here are all the ones I know of:

  • favicon.ico
    Gives your pages an icon in tabs, bookmarks, etc.
  • robots.txt
    Useful if you need to block any pages from search engines, but can be blank to allow access to everything. It can also contain the location for your sitemap.
  • Webmaster Tools verification files
    The easiest method of verification (saves having useless meta tags in your HTML code). There's google[hash].html and BingSiteAuth.xml. I have one for Yahoo in my root too but that's probably no longer needed.
  • crossdomain.xml
    This file is requested by Adobe Flash any time Flash needs to connect to your site. You will probably never see this file in your logs on a small site, but I had a bunch of requests for it on one site. You can just create a blank file since the default is to block access to external sites.
  • .htaccess (Apache)
    Obviously not requested by browsers/bots but vital for most sites.

Finally, regarding the icons that the iPhone insists on requesting: there are several sizes that may get requested. My iPhone 3G (iOS 5) requests the 57x57 version and I see 72x72 and 114x114 (most common) in my error logs. So if you want to eliminate all 404s from your logs, mind-boggingly you would need to provide all of these:

  • /apple-touch-icon-57x57-precomposed.png
  • /apple-touch-icon-57x57.png
  • /apple-touch-icon-72x72-precomposed.png
  • /apple-touch-icon-72x72.png
  • /apple-touch-icon-114x114-precomposed.png
  • /apple-touch-icon-114x114.png
  • /apple-touch-icon-precomposed.png
  • /apple-touch-icon.png
| improve this answer | |
  • 1
    .htaccess is specific to Apache and, even for those using Apache, it is a poor choice for performance if one has access to the VirtualHost definitions. – danlefree Dec 16 '11 at 0:42
  • +1 for crossdomain.xml - that hadn't occurred to me, and is a great idea to include. – Jacob Hume Dec 16 '11 at 2:36
  • I had once a bot requesting the sitemap.xml about once a day. And I never registered this file somewhere. So this bot was just guessing it. – Raffael Luthiger Dec 16 '11 at 19:25
  • @DisgruntledGoat: Thank you for the update. I didn't know that those other sizes exists too. – Raffael Luthiger Dec 19 '11 at 20:58
  • 1
    Silverlight also uses clientaccesspolicy.xml. – Lèse majesté Feb 15 '12 at 1:08

I think you probably have it.

Even apple touch icon is not entirely nescessary, since if you choose not to have it (and don't link to it) then iOS will just use a thumbnail of the page (personally, I add them in though).

I'd say you only really NEED favicon.ico and robots.txt, sitemap.xml is optional too.

| improve this answer | |
  • 1
    This wouldn't prevent the request, and thus the 404 errors that Raffael is seeing in their logs. If you were going to concentrate on which files make the most impact, I'd say you're right. – Jacob Hume Dec 16 '11 at 2:34

Another one that I've noticed is browserconfig.xml which is some bastardisation requested by IE. Bloody rubbish filling up the logs.

| improve this answer | |
  • Thanks. Do you know what should be in this file? – Raffael Luthiger Jan 8 '14 at 15:47

Newer apple versions request these files (in this order)

  • /apple-touch-icon-120x120-precomposed.png
  • /apple-touch-icon-120x120.png
  • /apple-touch-icon-precomposed.png
  • /apple-touch-icon.png
| improve this answer | |
  • Thanks. This would extend the list from DisgruntledGoat. – Raffael Luthiger Oct 9 '14 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.