2

I know that

  • Robots Exclusion Protocol (robots.txt)
  • Other similar .txt protocols (including sellers.json)
  • favicon.ico

must be placed in the root directory (according to their official spec), like this:

  • /ads.txt
  • /favicon.ico
  • /humans.txt
  • /robots.txt
  • /sellers.json

What about:

  • /.well-known/ ?

Must /.well-known/ also be placed in the root directory according to its official spec?

Are there any other files or folders which must be placed in the root directory?

10
  • 1
    Technically nothing has to be placed in the document root directory of your web server. Even if a document has to be served from the root of your URL, you can use server configuration such as Alias or RewriteRule to serve it from any directory of your choosing. Commented Sep 14, 2020 at 10:44
  • 1
    See also What is the .well-known/ directory and how should I set it up?. The .well-known RFC is an attempt to "fix" this "problem". In the future anybody that creates a URL that should be on every website should no longer put it at the root of the site, but rather prefix the URL path with /.well-known/. If people follow that RFC, there shouldn't be any additional URLs created that need to be at the root of your site. Commented Sep 14, 2020 at 10:48
  • This is definitely reassuring to learn, @StephenOstermiller. I'd wondered if it would be legitimate to 301 Redirect a robots.txt file using Apache .htaccess mod_rewrite or if search engines would simply ignore a 301 redirected protocol. Nevertheless, in this case, I'm unsure I can take advantage of redirection - my underlying issue is that I want to be able to set up one or more non-root site configurations in the same webspace. If there were only one I could certainly redirect from /robots.txt to /my-site-configuration/robots.txt... but if I have more...
    – Rounin
    Commented Sep 14, 2020 at 10:53
  • I also like the idea of using /.well-known/ for all standard configuration and protocol files. I really like this idea. But I can't find /.well-known/robots.txt in the wild anywhere.
    – Rounin
    Commented Sep 14, 2020 at 10:55
  • 1
    robots.txt was created before .well-known. The RFC for .well-known states that it only applies to URLs created in the future. URLs such as /robots.txt that existed beforehand will remain at their current root URL. Commented Sep 14, 2020 at 10:56

2 Answers 2

2

Must /.well-known/ also be placed in the root directory?

The /.well-known/ prefix is a defined standard (RFC8615), to which specific URI suffixes can be officially registered. So yes, it would need to reside in the root directory.

More info:
https://serverfault.com/questions/795467/for-what-is-the-well-known-folder

Others I can think of, off hand...

  • humans.txt
  • Google site HTML verification file: google123...html
  • Bing site HTML verification file: BingSiteAuth.xml
  • Other site verification files ... whilst for most this list is probably limited, if you include all possible 3rd party services then this list has potential to grow significantly?!

Of course, they don't necessarily need to physically exist in the site root, but they need to be accessible via the site root URL (if at all).

2
  • Thanks, @MrWhite. Just to clarify, are you stating it isn't the case that /.well-known/ must be in the root directory? If it can go elsewhere without breaking things then I'll be happy to remove it from the must be in the root directory list.
    – Rounin
    Commented Sep 13, 2020 at 23:53
  • No, it must be available in the root directory. The /.well-known/ prefix is a defined standard. Whilst this location can perhaps be configured for certain services, if you are on a shared server you will have less control over this. Sorry, my answer wasn't particularly clear, I'll update it...
    – MrWhite
    Commented Sep 14, 2020 at 0:50
1

I think it may be difficult to get an exhaustive list of all possible files which need to live at the root. For one thing, different content management systems may place various files at the root, while others may place those elsewhere, so it depends on what platform you're using.

Generally, your index.html file will be found at the root, but keep in mind that your root is still a folder, which may have different names, depending on your web host or your CMS.

Then, you have changing standards. The sitemap.xml file used to be commonly be placed at the root, but these days many CMS's like WordPress (via plugins like Yoast) allow for creation of a sitemap_index.xml file, which then leads to a list of sitemaps broken down by content type. Sometimes, they all live at the root; other times, they're in a directory. Having them in a directory is okay, as long as the sitemap index file is at the root and the search bots can easily find and crawl that directory. Thus, the sitemap.xml file may not exist on a website at all anymore, replaced by a (slightly) more complex sitemap information architecture. More on WordPress XML sitemaps here.

And then you have specific use cases. If your website is a publisher and actively sells inventory for ads to display on, you need an ads.txt file. This file should be at the root. If you're an ad exchange or an SSP (sell side platform), you need a sellers.json file, which should also live at the root. Read more about ads.txt and sellers.json.

Perhaps the best way to go about it is, learn more about your CMS, figure out the functionality you're looking for, and follow the standard, where the documentation will tell you where the crucial files should live.

For a look at how WordPress does it, here's a starter article.

6
  • 1
    Thanks, @HenryVisotski. To clarify, I'm not looking for "can be in the root directory" or "is usually found in the root directory" or even "should be in the root directory" - only "MUST be in the root directory".
    – Rounin
    Commented Sep 13, 2020 at 21:15
  • Re: "learn more about your CMS" - heh, I'm writing a CMS. That's why I'm asking the question at the top - to verify I've not missed any files or folders which have a fixed, pre-designated location.
    – Rounin
    Commented Sep 13, 2020 at 21:19
  • 1
    Writing a CMS? Now that's cool! @Rounin I'd say take a look at what WP and Drupal are doing, they're so well established that their lead is good to follow or to get ideas from. This is a good starting point for WP: wpbeginner.com/beginners-guide/… . Also, besides the index.html file, you'll want the .htaccess file at the root. Commented Sep 13, 2020 at 21:42
  • 1
    Link added to the response, @Rounin - thanks for the feedback! Yeah, if your CMS is different, then probably only the robots.txt and ads.txt, as that's the expected location if you're a bot. Ads.txt is now crucial if you are a publisher trying to monetize; I'm an ad tech eng so I can vouch for that. Favicon should be in the same folder as index file, but that's also optional; see here: stackoverflow.com/questions/21359342/… Commented Sep 13, 2020 at 22:25
  • 1
    "Then, you have changing standards. The sitemap.xml file used to be commonly be placed at the root" - The sitemap XML file never needed to be located in the site root and it never needed to be called "sitemap.xml" - that was simply a convenience that sites tended to adopt. Regardless of where it is or what it's called you still needed to inform search engines where to find it.
    – MrWhite
    Commented Sep 14, 2020 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.