I am currently using the Robots.txt file (the standard one of Joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.
Is there a reason to use one over the other?
Add a comment
|
3 Answers
The meta tags are sometimes useful as they offer more fine-grained guidelines for robots than robots.txt. They can also be used if you do not have access to the root directory of the server.
However, the robots.txt is more powerful, as it can be used to restrict access to entire directories (while the scope of a meta tag is always constrained to the single file it appears in).
If you use both, be aware that a well-behaved robot will not even see a meta tag if robots.txt disallow crawling the directory where it appear. I.e. there is nothing to be gained in adding meta tags (for finer resolution, etc.) to files that are in areas of the site disallowed by robots.txt. If you for some reason want to use meta tags, you must allow crawling of the directory they appear for them to be obeyed.
And while there is no harm done using both, there is no need to add meta tags if your robots.txt does what you want.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710
The noindex meta standard is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.
This tag is still useful and used by many plugins in WordPress, Joomla, Drupal and so on.. For example Yoast SEO for WordPress when you add a page or post you can click an option 'noindex' that inserts the noindex tag, this is incredibly easy and useful.
But I consider the use of both at the same time to be best.
answered Apr 26, 2013 at 11:47
-
If you use both robots.txt and robots meta tag to 'protect' the same page then the robots meta tag won't be seen since the robots.txt will prevent the page from being crawled.– MrWhiteCommented Apr 26, 2013 at 13:33
-
Not true, Robots prevents indexation not crawling, Ill try and find the GWT video that I seen about this when im home tonight. I could be wrong but im 99% sure that a GWT employee Said. Commented Apr 26, 2013 at 15:18
-
On the page you link to it states, "if you've used your robots.txt file to block this page, we won't be able to see the tag". In fact robots.txt does not necessarily prevent it from being indexed and appearing as a URL-only link in the SERPs if other pages are linking to it: webmasters.stackexchange.com/a/34171/1243– MrWhiteCommented Apr 26, 2013 at 18:45
-
Yep, your right but take this into account if you have a page that is linked too on your site that is disallow in robots then this page could be indexed, thats my point. For example your have a link at the top of the page that links to a login page that you have disallow in robots. because Google can discover this page it will appear indexed (not fully, but the link will be indexed). Commented Apr 26, 2013 at 21:30
The html meta index tags have been depreciated in favor of using a robots.txt
As for workflow the nice thing about the robots.txt file is that you can just update the txt file if you want to make a change, so you only have to make it once, for example if you wanted to noindex the whole site this could be done with 2 lines of code, were as with the meta tags you have to do it on each page, so its more complicated to maintain.
-
1The HTML META index tags have not been deprecated in favor of using a robots.txt - they serve a related but different purpose. A robots.txt file prevents pages from being crawled, but can still be indexed if pages link to the page in question. A robots META tag (or corresponding X-Robots-Tag HTTP response header) on the other hand prevents the page from being indexed.– MrWhiteCommented Apr 26, 2013 at 13:25
-