I did a quick search for how to create a robots.txt file and found these two relevant links
Normally answers here should not be "link only" but this is such basic stuff and there are ...
I want to change access permission of robots.txt file to 403. Just my static ip can access directly not any one.
This doesn't make sense. The only reason to have a robots.txt in the first place is to have it publicly accessible for search engines to determine if there are any URLs that should not be crawled.
Blocking robots.txt with a 403 HTTP response is ...