After reading up on this Google Webmasters article on creating a robots.txt file, I noticed that they mentioned that the robots.txt file should always be at the root directory (/). However, I've been getting accustomed to putting all of my resource files (CSS, JS, images, PDFs, etc) in a static files sub-directory (/static/...) and perma-redirecting (301) associated links to them (in the case of meta-tags, etc).

  1. Does my solution of redirecting /robots.txt to /static/file/robots.txt function as expected?

  2. Even if my solution does work, should I still bite the bullet and do the proper thing and just serve the file as the spec tells me to do?

  • Why do you typically use a "static" directory? I would never recommend doing that. Usless directories that make URLs more complicated are not good usability. – Stephen Ostermiller Apr 19 '20 at 8:29
  • Where else would you put static resources? This seems to be the standard solution for web applications built in Node/Express, Go(lang), etc. – goddtriffin Apr 19 '20 at 20:03
  • I put everything in the web root directory, I don't see a reason to separate files in directories by filetype because that is what extensions are for. – Stephen Ostermiller Apr 19 '20 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.