3

I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create one in cPanel. My friend told me it can also make my website vulnerable to attack.

2
  • 1
    I'll provide a comprehensive answer here but it's going to be long form and it's going to take some time. Dec 22, 2021 at 23:47
  • "My friend told me it can also make my website vulnerable to attack." You may want to change friends, that one is giving you incorrect information. Existence or absence of this file, and its content, has no impact on your attack surface. Apr 21 at 1:04

1 Answer 1

1

I did a quick search for how to create a robots.txt file and found these two relevant links

Google: https://developers.google.com/search/docs/advanced/robots/create-robots-txt

Bing: https://www.bing.com/webmasters/help/how-to-create-a-robots-txt-file-cb7c31ec

Normally answers here should not be "link only" but this is such basic stuff and there are so many sites with the information, it is easy to find.

3
  • 7
    Even when many other sources are available, it is still worth it to create answers here rather than just providing links. If this question were to get a bunch of traffic, the user experience would be frustrating if everybody has to click. Jul 26, 2021 at 10:59
  • @StephenOstermiller Looks like a prime question for our "general question" initiative. Dec 22, 2021 at 23:43
  • If giving links on this subject, you might as well use robotstxt.org to not depend on any specific search engine. Apr 21 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.