6

While the German Wikipedia claims that all is a valid value, I've no idea where that information comes from, and German Wikipedia does not cite any source. It is AFAIK wrong, and IMHO a good example for why one should not rely on Wikipedia for this type of information. If you want to use the robot meta tag to direct googlebot, the best information about ...


6

No. Google may sometimes execute Javascript to determine content, but it's a very bad idea to rely on Javascript for your site to be crawled as you want it to be.


5

The default behaviour of robots if there is no robots.txt and robots meta tag is to follow links on the page and index it. That's why <meta name="robots" content="index, follow" /> is not necessary if you want the default behaviour of robots for your page.


5

It is more or less the same as writing: <meta name="robots" content="index, follow"> Interestingly, the "all" variant is not actually suggested by Google as an alternative, but it works* nonetheless. Either command (when placed in the <head> section of your HTML code) tells search engines to index the page the tag ...


4

The behaviour for two conflicting meta tags isn't defined, as far as I know, but most likely the most restrictive rule will win. This happens the similar case of robots.txt file vs meta robots tag. If robots.txt prevents indexing a page and meta-robots doesn't, the page will not be indexed. And if robots.txt allows a page but meta-robots blocks it, it will ...


3

Per these tests by adding robots meta tag will work: https://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157 (Adam Audette on May 8, 2015 at 12:32 pm).


3

The nosnippet tag does prevent caching, but it also prevents a snippet from showing up under the clickable title of you page in SERPs, thus basically devaluing your carefully written descriptions and giving them no chance to show up, correctly or otherwise. If you want to prevent caching, I recommend using noarchive instead, which will do just that, and only ...


3

There would seem to be only 8 mainstream directives for the robots meta tag (or X-Robots-Tag HTTP response header) that really matter. Google includes 9 directives in its list of valid directives, including all, which is arguably just ignored. Note that Google does not include index and follow in its list of supported directives. These are most probably ...


3

option 3. Nothing is by far the best approach. If you mean that the robots tag should be omitted entirely, then yes, that would be the best option if you want the page to be indexed and followed. The other options just add superfluous bytes and will be ignored. As mentioned in my answer to your other related question, Google only includes all (out of what ...


3

Google will still crawl the page, but if there is information about your page on dmoz, Google wont display that information on it's search result page. One source Google uses to generate snippets is the Open Directory Project. You can direct us not to use this as a source by adding a meta tag to your pages. To prevent all search engines (that ...


3

The same as rel="nofollow" in a link (apart from the fact the the meta tag would be page wide rather than on a per link basis) Yes, same as rel="nofollow", except the nofollow in the meta tag applies to all the links on the page. Or does the meta tag version refer to wether robots should crawl the links on the page, rather than if they pass 'link juice'? ...


3

The meta tags are sometimes useful as they offer more fine-grained guidelines for robots than robots.txt. They can also be used if you do not have access to the root directory of the server. However, the robots.txt is more powerful, as it can be used to restrict access to entire directories (while the scope of a meta tag is always constrained to the single ...


2

Where does it add that ? I believe you looked in the iframe that the Adsense code generates. If this is the case, then your website won't be affected. They add that because they don't want search engines to crawl their ads or index their iframe contents.


2

Blocking images from Google Use any of the two tags, <meta name="robots" content="noimageindex"> Or, <meta name="googlebot" content="noimageindex"> Note that the images on the page may still be included in the image index if they are linked to by other pages. See the below links. https://support.google.com/webmasters/answer/79892?hl=en ...


2

Try this for the page you need to ignore indexing of images (by adding the following meta tags): <meta name="robots" content="noimageindex"> This tag will prevent indexing images from your site or external site on that page.


2

no bot will ever reach a 'logged' page, as [i think] it can only be obtained filling the login form; thing that no robot [AFAIK] can do today


2

I tested this approach - doesn't worked for me. But its only my experience. John Mueller means, it should work. I wouldn't noindex categories, where all products are sold out. I would 410 them only if there will be never new products in the category. But even if a category is temporarily empty - let it be indexed, publish a notice for users and offer some ...


2

If you "dynamically add that meta tag" for only the first URL then it will not affect indexing of the second. Whilst it might be the same file (product.aspx) that processes both requests, they are two different URLs.


2

Faceted navigation / search filtering allows visitors to narrow their search for specific products. Such navigation features generate crawl paths that are of no real value to a search engine but are useful for humans. By adding meta noindex,follow to these pages it instructs the search engine to crawl into the product pages for indexing but keeps the faceted ...


2

By default Google or any other modern search engine will crawl, index and follow the links in a page when the above meta tag is absent. In your example Google will find the noindex instruction and this is what is expected to happen: Google bot will crawl the page (if robots txt allows it) Your page will not get indexed and will be removed from SERP, ...


2

Once a page is out there, Google is just going to crawl it, suck it up, and retain it in its vast and very long term memory. If the page you want removed is in Google's SERPs, then the noindex alone won't resolve this, it'll simply advise new search engines not to rank it. You need to add nofollow,noarchive to that page's robots command (it can all be in ...


2

Just go to your C panel>>public_html>>site directory and create a file named "robots.txt" in it paste the following code User-Agent: * Disallow: /privacy.php/ Disallow: /terms.php/ Allow: Sitemap: http://www.xxxxxxx.xxx/sitemap.xml Change the links above to match up with your site. With the above, google bots can never crawl any information in the "...


1

You cannot redirect using .htaccess and also affect the title and meta description of the resulting page. .htaccess can rewrite URLs and take you to different places, but it can't change tags on the destination page. The example you provided is likely using a meta refresh or some other type of redirect, so visitors see the old page briefly, and are then ...


1

Adding a meta tag and disallowing in robots.txt if not a good approach. You should only do one or the other. If you disallow then Google won't be able to see the status when it crawls. Of the two, the meta tag is a much stronger signal than robots.txt. Google sometimes indexes pages that are disallowed. However if Google can crawl and see the tag it won'...


1

I would say (global) *?*foo* or (only /tool/input) /tool/input?*foo* please note: parameters are exchangeable and have no sorting!


1

404 isn't ideal. Serving 410 error code is better as it indicates permanent removal. Also Adding nofollow/noindex meta tags/robots.txt on those bad URL's is a good way to speed things up. Serving correct error codes, meta data, redirecting, disavowing incoming links would all be viable process for different reasons. Disvow will help google understand ...


1

The tag tells the search engines not to include the page in their listings. As far as i'm aware you can't stop a search engine crawling your page as it will need to craw the page in order to see if you remove the tag. (If you change your mind and want the page to be listed)


1

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710 The noindex meta standard is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis. This tag is still useful and used by many plugins in WordPress, Joomla, Drupal and so on.. For example Yoast SEO for ...


Only top voted, non community-wiki answers of a minimum length are eligible