3

I really like the idea of using .htaccess rewrite rules in combination with the /.well-known/ folder for keeping my webspace tidy and coherently / consistently organised.

For instance, I know that webcrawlers (and humans)

  • will look for my robots.txt in the root folder; and
  • will look for my security.txt in the /.well-known/ folder.

This means that files which ought to be near each other would, conventionally, be separated.

The Setup

But I also know I can rewrite requests using .htaccess, such that I can rewrite a request for:

  • /robots.txt to /.well-known/protocols/robots.txt
  • /.well-known/security.txt to /.well-known/protocols/security.txt

Great. Now all the protocols:

and maybe even:

can live together.

But what about sitemap.xml and serviceworker.js ?

So far, so good.

But what if, analagously, I want to have something like:

  • /.well-known/sitemaps/sitemap.xml
  • /.well-known/serviceworkers/serviceworker.js

I know I can use .htaccess to rewrite requests for /sitemap.xml and /serviceworker.js, but I also know that these files are location-sensitive.

That is, the directives in each of these files are only supposed to apply to files:

  • in the same folder; and
  • in subfolders of that folder

See:

The location of a Sitemap file determines the set of URLs that can be included in that Sitemap. A Sitemap file located at http://example.com/catalog/sitemap.xml can include any URLs starting with http://example.com/catalog/ but can not include URLs starting with http://example.com/images/.

Source: https://www.sitemaps.org/protocol.html#location

and:

The service worker will only catch requests from clients under the service worker's scope. [...] The max scope for a service worker is the location of the worker.

Source: https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers

But what is the sensitive location in this context?

Is it the "location" as it appears in the filepath request, or is it the actual filesystem location?

8
  • 3
    While there is merit to the argument that similar files should live in the same place, it is easier for somebody to understand what is going on on your server when files are found in their default locations. If somebody new comes in and wants to add to your robots.txt they may have trouble locating it or try to create a new one assuming that one does not exist. Jun 28, 2021 at 18:53
  • 2
    None of these files might exist on the server's file system, they could all be created and served directly by the web server software. Jun 28, 2021 at 23:24
  • 1
    Excellent point, @curiousdannii. I've now got my head around the idea that the key requirement in all of this (for certain documents like /robots.txt and /.well-known/security.txt) is only a correctly formatted URL Path. As such - and fairly ironically, given that I was originally setting out to find a way to put all protocol files and other meta documents in the /.well-known/ folder - I've now ditched the /.well-known/ folder entirely and moved all the meta documents into subfolders like /.assets/theme/meta/protocols/ and /.assets/theme/meta/sitemaps/.
    – Rounin
    Jun 29, 2021 at 9:04
  • 1
    The sitemap rules you mention are not true in practice. You can put your sitemap anywhere and refer to any URLs in any directory (on the same domain). Jul 3, 2021 at 9:26
  • 1
    @Rounin I don’t know exactly what would count as “evidence”. I thought there was a question on here but I can’t find it now. But you can just put a sitemap in a random directory and submit it to Google Search Console and it will process it just fine. Jul 4, 2021 at 21:42

1 Answer 1

4

Is it the location in the requested filepath, or the actual filesystem location?

The location is with regards to the requested URL-path (not strictly a "filepath"). The location of the file on the underlying server's filesystem is irrelevant and unknown to the user-agent making the request.

All requests to these files are client-side HTTP requests (ie. URLs). The same as all the other file types you've mentioned.

UPDATE:

That is, the directives in each of these files are only supposed to apply to files:

  • in the same folder;
  • and in subfolders of that folder

What is perhaps confusing here is the use of the term "folder". It's not really a "folder" (which implies a physical directory on a filesystem), it's a "URL-path" - what you see in the browser's address bar. Unfortunately, "folder", "directory" and "file-path" get used a lot when describing URLs, but that is strictly incorrect. When you rewrite the request in .htaccess, you are rewriting the request from a URL-path to a file-path.

5
  • Thank you, @MrWhite. So, for clarification, a sitemap.xml at https://example.com/.well-known/sitemaps/sitemap.xml can include any URLs starting with https://example.com/catalog/ (even though the sitemap in question resides in https://example.com/.well-known/sitemaps/) so long as that sitemap is requested from https://example.com/catalog/sitemap.xml?
    – Rounin
    Jun 28, 2021 at 14:49
  • 1
    @Rounin Yes, that's right.
    – MrWhite
    Jun 28, 2021 at 14:58
  • Great. Thanks very much for that clarification.
    – Rounin
    Jun 28, 2021 at 15:07
  • 1
    @Rounin You're welcome. Some of this confusion might come about from using the term "folder" to refer to a URL-path segment. I've updated my answer.
    – MrWhite
    Jun 28, 2021 at 15:33
  • 2
    @Rounin Consumers of these files have no way of knowing about your rewrite rules or where these files are stored on disk. They can only know about the URL path from which they are requested. The reason that the restrictions are relative to the URL path is that all the the requirements are enforced by the client and the client has no way to know anything except the URL path. Jun 28, 2021 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.