Should I use both Robots.txt and META NAME="ROBOTS"?

Question

I am currently using the Robots.txt file (the standard one of Joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.

Is there a reason to use one over the other?

Answer 1

The meta tags are sometimes useful as they offer more fine-grained guidelines for robots than robots.txt. They can also be used if you do not have access to the root directory of the server.

However, the robots.txt is more powerful, as it can be used to restrict access to entire directories (while the scope of a meta tag is always constrained to the single file it appears in).

If you use both, be aware that a well-behaved robot will not even see a meta tag if robots.txt disallow crawling the directory where it appear. I.e. there is nothing to be gained in adding meta tags (for finer resolution, etc.) to files that are in areas of the site disallowed by robots.txt. If you for some reason want to use meta tags, you must allow crawling of the directory they appear for them to be obeyed.

And while there is no harm done using both, there is no need to add meta tags if your robots.txt does what you want.

Answer 2

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710

The noindex meta standard is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.

This tag is still useful and used by many plugins in WordPress, Joomla, Drupal and so on.. For example Yoast SEO for WordPress when you add a page or post you can click an option 'noindex' that inserts the noindex tag, this is incredibly easy and useful.

But I consider the use of both at the same time to be best.

Answer 3

The html meta index tags have been depreciated in favor of using a robots.txt

As for workflow the nice thing about the robots.txt file is that you can just update the txt file if you want to make a change, so you only have to make it once, for example if you wanted to noindex the whole site this could be done with 2 lines of code, were as with the meta tags you have to do it on each page, so its more complicated to maintain.