I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are:
- Adding it into the robots.txt file:
- Adding a meta tag:
<meta name="robots" content="noindex, nofollow">
Which method would work the best? I want this directory to remain "invisible" from search engines so it does not affect any of my site's ranking.
In other words, I want this directory to be neutral/invisible and "just there." I don't want it to affect any ranking. Which method would be the best to achieve this?