In my Webmaster's Tools crawl errors I have a few access denied locations:


I had a look at RFC 5785 but I didn't really understand it. I actually struggle to follow a lot of RFC documents, they seem to be filled with a lot of unexplained and ambiguous terminology.

What is the purpose of this directory and these three files? What should their contents be? Should I even worry about this? Should I block the directory in robots.txt?

This is the first I've ever come across this directory, I don't believe it exists and I have made no attempt to populate it (which is probably why it is denied). I also am unaware of any links to this directory from my site.

  • 1
    Denied locations are normally a result of blocked resources in either the your robots.txt or htaccess file. Please edit your question and include the contents of these files, additionally... does .well-known/ exist? Jul 12 '16 at 15:57
  • 4
    I actually struggle to follow a lot of RFC documents, they seem to be filled with a lot of unexplained and ambiguous terminology. You are not alone! Cheers!!
    – closetnoc
    Jul 12 '16 at 15:58
  • 1
    If I have not gone cross-eyed, .well-known/ should be a hidden directory and not be available to external requests, however, available to server side coded elements such as CGI programs, Java, and the like.
    – closetnoc
    Jul 12 '16 at 16:01
  • 2
    It's likely picked the link up from another page. It's an attempt at a junk URI, I wouldn't be worried unless you have resources there and someone knows about them.
    – L Martin
    Jul 12 '16 at 16:08
  • 1

RFC 5785 can be paraphrased:

If you create a new URL such as robots.txt or favicon.ico that is expected to be at a certain place on every website, you shall henceforth make such a URL start with /.well-known/.

Furthermore, you will register all such URLs with the IETF so that there is a big central list of all these URLs.

The registry appears to be here: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml You can look through the registry, decide if any of the requested URLs serve a purpose that would help your site, and create documents if some would be useful.

Of the URLs that you mention in your question, only assetlinks.json appears to be registered properly.

There is generally no reason to create documents just because Google reports them as 404. Google expects to get 404 errors on sites. They even consider it a sign of a healthy site that URLs without documents return the correct status (404).

  • 1
    Ah ok, so to comply, technically my robots.txt should be moved to a publicly accessible location www.mysite.com/.well-known/robots.txt rather than just being at www.mysite.com/robots.txt? But ultimately it doesn't matter for now?
    – Arth
    Jul 12 '16 at 16:17
  • 8
    No, I wouldn't move robots.txt. That URL was created well before this .well-known edict and I expect that it will remain grandfathered into it's usual location. Jul 12 '16 at 16:18
  • 1
    Cool, that makes sense.. thanks for the help! If you could paraphrase all the other RFCs too that'd be swell :)
    – Arth
    Jul 12 '16 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.