I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are:

  1. Adding it into the robots.txt file: Disallow: /directory/
  2. Adding a meta tag: <meta name="robots" content="noindex, nofollow">

Which method would work the best? I want this directory to remain "invisible" from search engines so it does not affect any of my site's ranking.

In other words, I want this directory to be neutral/invisible and "just there." I don't want it to affect any ranking. Which method would be the best to achieve this?


Supposedly only robots.txt will be fine, but that depends on how the crawler bot is programmed. Almost all 'legal' crawlers (If not all) use robots.txt, and for disabling a directory robots.txt is the best option, as you don't have to add a meta tag to each file, and also disallows txt files, images and everything you forgot/can't add a meta tag and may be linked from elsewhere. Also, if you upload more files in the future to the directory, you don't have to worry about remembering about adding meta tags too.

Definitely is safer to use robots.txt to deny the access to an entire directory.

EDIT: Although Disallow tells robots to disallow a directory from being crawled, maybe Google indexes it, as stated here: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h17 so using the tag instead of the robots.txt will be better.

However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant.

| improve this answer | |
  • Awesome! Also, by adding this to the robots.txt file, will it make the directory "invisible" to the search engine? Meaning will it not affect the ranking of the other pages on my site. – Gkhan14 Jun 30 '13 at 18:09
  • Wait, I'm adding more info. Just read the Google faq and his bot is a bit special with robots.txt. – Jorge Fuentes González Jun 30 '13 at 18:13
  • And, in reply about the pagerank, that depends on how Google calculates pagerank exactly and they don't tell much about that when talking about bad practices, but seems that having 'Duplicate content' (Google hates that so much) don't affects your pagerank as long as the page has the nonindex tag, as you can read here: support.google.com/webmasters/answer/66359?hl=en so maybe that works with other practices. – Jorge Fuentes González Jun 30 '13 at 18:24
  • Thanks! Lastly, do you think it's safe to just place a 404 error for the entire directory in my .htaccess file, according to the IP? This directory is mainly just for me and a few other people. – Gkhan14 Jun 30 '13 at 18:35
  • What do you mean with safe? .htaccess can only be edited by the admin/admins that has write access to the file so any normal user can edit it, is as safe as your config files. If you like the main answer, please, accept it by clicking the 'V' at his left please :P – Jorge Fuentes González Jun 30 '13 at 18:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.