4 votes
Accepted

Are there only 12 directives for <meta name="robots" />?

There would seem to be only 8 mainstream directives for the robots meta tag (or X-Robots-Tag HTTP response header) that really matter. Google includes 9 directives in its list of valid directives, ...
DocRoot's user avatar
  • 4,297
4 votes
Accepted

Can the content attribute of <meta name="robots" /> be left empty?

option 3. Nothing is by far the best approach. If you mean that the robots tag should be omitted entirely, then yes, that would be the best option if you want the page to be indexed and followed. ...
DocRoot's user avatar
  • 4,297
3 votes
Accepted

What are the potential side effects of the "nosnippet" robots meta tag?

The nosnippet tag does prevent caching, but it also prevents a snippet from showing up under the clickable title of you page in SERPs, thus basically devaluing your carefully written descriptions and ...
Henry Visotski's user avatar
3 votes
Accepted

robots meta tag for specific web crawler

<meta name="robots" content="index, follow"> <meta name="googlebot" content="index, follow, noimageindex"> Yes, this will "work" as ...
MrWhite's user avatar
  • 42.8k
3 votes

What is a difference in the order of "follow" and "index" in a robots meta tag?

There is no difference between follow, index and index, follow. These would both permit robots to index content and follow links*1. However, this is the default action for all bots, so the element is ...
MrWhite's user avatar
  • 42.8k
3 votes

What does it mean, 404 page with noindex and non-self-referencing canonical?

In my opinion, Meta Robots and Canonical should be removed. What would you think? The meta robots (noindex) tag is unnecessary, but it can stay. Providing the response has a 404 HTTP status then ...
MrWhite's user avatar
  • 42.8k
3 votes
Accepted

does google crawl/follow link on a page with meta robots noindex

By default Google or any other modern search engine will crawl, index and follow the links in a page when the above meta tag is absent. In your example Google will find the noindex instruction and ...
Raul Reyes's user avatar
  • 2,476
3 votes
Accepted

Meta tag robots noodp

Google will still crawl the page, but if there is information about your page on dmoz, Google wont display that information on it's search result page. One source Google uses to generate snippets ...
RolfRB's user avatar
  • 363
3 votes
Accepted

difference between `<meta name="robots" content="nofollow" />` and `rel="nofollow"`

The same as rel="nofollow" in a link (apart from the fact the the meta tag would be page wide rather than on a per link basis) Yes, same as rel="nofollow", except the nofollow in the meta tag applies ...
MrWhite's user avatar
  • 42.8k
3 votes

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

Per these tests by adding robots meta tag will work: https://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157 (Adam Audette on May 8, 2015 at 12:32 pm).
kurt lindner lippisch's user avatar
2 votes

Website not properly indexed by Google search console says "'noindex' detected in 'robots' meta tag" while this is not the case

"index" is not an approved value for use in a robots meta tag. See: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag Instead you should use "all". <...
Stephen Ostermiller's user avatar
2 votes

Use robots.txt to prevent privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google

Just go to your C panel>>public_html>>site directory and create a file named "robots.txt" in it paste the following code User-Agent: * Disallow: /privacy.php/ Disallow: /terms.php/ Allow: Sitemap: ...
Yesokay's user avatar
  • 21
2 votes

does google crawl/follow link on a page with meta robots noindex

Once a page is out there, Google is just going to crawl it, suck it up, and retain it in its vast and very long term memory. If the page you want removed is in Google's SERPs, then the noindex alone ...
Andrew Martin's user avatar
2 votes

What does it mean, 404 page with noindex and non-self-referencing canonical?

The definition of 404 is 404 Not Found The requested resource could not be found but may be available in the future. Subsequent requests by the client are permissible. So you should be generating ...
Rohit Gupta's user avatar
  • 2,466
1 vote
Accepted

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

As discussed in the comments, you should disallow your entire site search results. However, there may be other pages for which you want to have search bots ignore some parameters. There are two ...
Stephen Ostermiller's user avatar
1 vote

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

It's recommended best practice to disallow indexing of Site Search Pages. Note that while this can be done in robots.txt, using: User-agent: * Disallow: /search/ where /search/ is the folder ...
Rounin's user avatar
  • 2,335
1 vote

Ask How To Seting httaccses meta desc title same with oldies web

You cannot redirect using .htaccess and also affect the title and meta description of the resulting page. .htaccess can rewrite URLs and take you to different places, but it can't change tags on the ...
WebElaine's user avatar
  • 1,920
1 vote

Disallowing URL parameter in robots.txt

Adding a meta tag and disallowing in robots.txt if not a good approach. You should only do one or the other. If you disallow then Google won't be able to see the status when it crawls. Of the two, ...
Stephen Ostermiller's user avatar
1 vote

Disallowing URL parameter in robots.txt

I would say (global) *?*foo* or (only /tool/input) /tool/input?*foo* please note: parameters are exchangeable and have no sorting!
Marius's user avatar
  • 199
1 vote

does google crawl/follow link on a page with meta robots noindex

The tag tells the search engines not to include the page in their listings. As far as i'm aware you can't stop a search engine crawling your page as it will need to craw the page in order to see if ...
Matthew Jasek's user avatar
1 vote

Method for removal of high number of spammy links

404 isn't ideal. Serving 410 error code is better as it indicates permanent removal. Also Adding nofollow/noindex meta tags/robots.txt on those bad URL's is a good way to speed things up. Serving ...
Randomer11's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible