I have a website that I wanna publish it... but for some security reason we want to make the website only accessible for those who has the link. which means prevent any kind of indexing or crawlers to index it so that it can not be founded from search engines. Could you help with some practical whys? I heard about robots.txt, but i don't know how to use it and where to put it on my host? and also I heard that there are a nonindex meta tag.
This is an example of security through obscurity. This is not a good practice. But if you must, then the noindex tag is respected by major search engines (Google, Bing, etc.).
But the better solution would be to:
- Password it, or
- Only allow certain IPs (your stakeholders), or
- Firewall/Fail2Ban any traffic with "bot" in it