I have a website that I wanna publish it... but for some security reason we want to make the website only accessible for those who has the link. which means prevent any kind of indexing or crawlers to index it so that it can not be founded from search engines. Could you help with some practical whys? I heard about robots.txt, but i don't know how to use it and where to put it on my host? and also I heard that there are a nonindex meta tag.

  • 1
    If it's truly a security concern you should use a password for access to the website. Aug 30, 2020 at 19:28
  • It will be better in my case to prevent it from anyone can find it in search engines. not to use the password because the site will be accessed by others.
    – Hunar
    Aug 30, 2020 at 19:35
  • 1
    we also have a special login page, but no signup page.
    – Hunar
    Aug 30, 2020 at 19:40
  • 2
    That sound like good security. If everything is behind login, search engines won't index any of it already. Aug 30, 2020 at 19:41
  • 1
    They can't index the contents of those pages because their crawlers can't see the contents. Google occasionally indexes URLs that it can't crawl because they are blocked by robots.txt but I've never seen Google index a URL that redirects to a login page. Aug 30, 2020 at 20:05

2 Answers 2


You can not prevent an URL from being indexed completely by any search engine or crawlers if it is publicly accessible.

  • Search engines are very good about honoring the noindex meta tag on publicly accessible pages. I don't think your answer is correct. Aug 31, 2020 at 9:47
  • In this case, it sounds like the website is password protected and not publicly accessible anyway. Aug 31, 2020 at 9:47
  • @StephenOstermiller Reputable search engines will honour noindex and robots.txt, but these methods rely entirely on the integrity of the crawler - they don't stop unscrupulous actors from accessing the data. As you say, they need additional security if they want to prevent unwanted access and in this case it sounds like they have, but I just wanted to highlight this for any other users who come across this question... additional security is necessary to prevent the pages being accessed. Aug 31, 2020 at 21:32
  • Yes, most search engines will honour noindex and robots.txt, but not all. And the question is clear about it: "completely by any search engine". So noindex is not sufficient? Sep 1, 2020 at 11:20
  • @StephenOstermiller "the website is password protected" Is it? As long as this is not clear, imho my answer is correct? But yes, password protection is an obvious solution. Sep 1, 2020 at 11:36

This is an example of security through obscurity. This is not a good practice. But if you must, then the noindex tag is respected by major search engines (Google, Bing, etc.).

But the better solution would be to:

  1. Password it, or
  2. Only allow certain IPs (your stakeholders), or
  3. Firewall/Fail2Ban any traffic with "bot" in it

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.