1

I want to change access permission of robots.txt file to 403. Just my static ip can access directly not any one. This position is true? why some people do that?

if it is true, What is the affect of this changes in my website seo and pages and other things? if it is false, tell why? please answer with your experience and reference.

1
  • 2
    "Just my static ip can access directly not any one." - Why do you want to do this? "This position is true? why some people do that?" - Oh, are you asking why?? Where did you get the idea for this? Which "some people do" this??
    – MrWhite
    Jul 22 at 14:48
1

I want to change access permission of robots.txt file to 403. Just my static ip can access directly not any one.

This doesn't make sense. The only reason to have a robots.txt in the first place is to have it publicly accessible for search engines to determine if there are any URLs that should not be crawled.

Blocking robots.txt with a 403 HTTP response is effectively the same as not having a robots.txt file. ie. You are permitting unrestricted crawl access to your site.

If you don't want a robots.txt file (ie. you want robots to crawl everything) then simply don't include a robots.txt file and allow requests to 404. Alternatively, create an empty (or minimal) robots.txt to prevent a plethora of 404s in your access logs.

If you want to prevent crawling of certain URLs on your site then you need a robots.txt file and it needs to be publicly accessible. Access to this file should not be restricted.

why some people do that?

I'm curious where you got the idea of this from?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.