0

I have a website that I wanna publish it... but for some security reason we want to make the website only accessible for those who has the link. which means prevent any kind of indexing or crawlers to index it so that it can not be founded from search engines. Could you help with some practical whys? I heard about robots.txt, but i don't know how to use it and where to put it on my host? and also I heard that there are a nonindex meta tag.

7
  • 1
    If it's truly a security concern you should use a password for access to the website. – Stephen Ostermiller Aug 30 '20 at 19:28
  • It will be better in my case to prevent it from anyone can find it in search engines. not to use the password because the site will be accessed by others. – Hunar Aug 30 '20 at 19:35
  • For security, every person who uses it should have their own username and password then. – Stephen Ostermiller Aug 30 '20 at 19:38
  • 2
    That sound like good security. If everything is behind login, search engines won't index any of it already. – Stephen Ostermiller Aug 30 '20 at 19:41
  • 1
    They can't index the contents of those pages because their crawlers can't see the contents. Google occasionally indexes URLs that it can't crawl because they are blocked by robots.txt but I've never seen Google index a URL that redirects to a login page. – Stephen Ostermiller Aug 30 '20 at 20:05
1

You can not prevent an URL from being indexed completely by any search engine or crawlers if it is publicly accessible.

7
  • Search engines are very good about honoring the noindex meta tag on publicly accessible pages. I don't think your answer is correct. – Stephen Ostermiller Aug 31 '20 at 9:47
  • In this case, it sounds like the website is password protected and not publicly accessible anyway. – Stephen Ostermiller Aug 31 '20 at 9:47
  • @StephenOstermiller Reputable search engines will honour noindex and robots.txt, but these methods rely entirely on the integrity of the crawler - they don't stop unscrupulous actors from accessing the data. As you say, they need additional security if they want to prevent unwanted access and in this case it sounds like they have, but I just wanted to highlight this for any other users who come across this question... additional security is necessary to prevent the pages being accessed. – FluffyKitten Aug 31 '20 at 21:32
  • Yes, most search engines will honour noindex and robots.txt, but not all. And the question is clear about it: "completely by any search engine". So noindex is not sufficient? – CE WebDesign München Sep 1 '20 at 11:20
  • @StephenOstermiller "the website is password protected" Is it? As long as this is not clear, imho my answer is correct? But yes, password protection is an obvious solution. – CE WebDesign München Sep 1 '20 at 11:36
1

This is an example of security through obscurity. This is not a good practice. But if you must, then the noindex tag is respected by major search engines (Google, Bing, etc.).

But the better solution would be to:

  1. Password it, or
  2. Only allow certain IPs (your stakeholders), or
  3. Firewall/Fail2Ban any traffic with "bot" in it

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.