Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.

Is there a reason to use one over the other?

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

The meta tags are sometimes useful as they offer more fine-grained guidelines for robots than robots.txt. They can also be used if you do not have access to the root directory of the server.

However, the robots.txt is more powerful, as it can be used to restrict access to entire directories (while the scope of a meta tag is always constrained to the single file it appears in).

If you use both, be aware that a well-behaved robot will not even see a meta tag if robots.txt disallow crawling the directory where it appear. I.e. there is nothing to be gained in adding meta tags (for finer resolution, etc.) to files that are in areas of the site disallowed by robots.txt. If you for some reason want to use meta tags, you must allow crawling of the directory they appear for them to be obeyed.

And while there is no harm done using both, there is no need to add meta tags if your robots.txt does what you want.

share|improve this answer
add comment

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710

The noindex meta standard is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.

This tag is still useful and used by many plugins in WordPress, Joomla, Drupal and so on.. For example Yoast SEO for WordPress when you add a page or post you can click an option 'noindex' that inserts the noindex tag, this is incredibly easy and useful.

But I consider the use of both at the same time to be best.

share|improve this answer
    
If you use both robots.txt and robots meta tag to 'protect' the same page then the robots meta tag won't be seen since the robots.txt will prevent the page from being crawled. –  w3d Apr 26 '13 at 13:33
    
Not true, Robots prevents indexation not crawling, Ill try and find the GWT video that I seen about this when im home tonight. I could be wrong but im 99% sure that a GWT employee Said. –  bybe Apr 26 '13 at 15:18
    
On the page you link to it states, "if you've used your robots.txt file to block this page, we won't be able to see the tag". In fact robots.txt does not necessarily prevent it from being indexed and appearing as a URL-only link in the SERPs if other pages are linking to it: webmasters.stackexchange.com/a/34171/1243 –  w3d Apr 26 '13 at 18:45
    
Yep, your right but take this into account if you have a page that is linked too on your site that is disallow in robots then this page could be indexed, thats my point. For example your have a link at the top of the page that links to a login page that you have disallow in robots. because Google can discover this page it will appear indexed (not fully, but the link will be indexed). –  bybe Apr 26 '13 at 21:30
add comment

The html meta index tags have been depreciated in favor of using a robots.txt

As for workflow the nice thing about the robots.txt file is that you can just update the txt file if you want to make a change, so you only have to make it once, for example if you wanted to noindex the whole site this could be done with 2 lines of code, were as with the meta tags you have to do it on each page, so its more complicated to maintain.

share|improve this answer
1  
The HTML META index tags have not been deprecated in favor of using a robots.txt - they serve a related but different purpose. A robots.txt file prevents pages from being crawled, but can still be indexed if pages link to the page in question. A robots META tag (or corresponding X-Robots-Tag HTTP response header) on the other hand prevents the page from being indexed. –  w3d Apr 26 '13 at 13:25
    
i stand corrected –  sam Apr 26 '13 at 15:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.