Questions tagged [meta-robots]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
169 views

Should Feed/RSS Be Blocked from Robots.txt?

Should Feed/RSS be blocked via robots.txt? Disallow: /*feed* Disallow: /*rss* If yes, is this the correct way to apply? If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
1 vote
2 answers
62 views

What does it mean, 404 page with noindex and non-self-referencing canonical?

I am confronted with a strange setup I never seen before - and I don't know, how to deal with it. Details: There is a bunch of no longer existing pages. All of them have Status code 404, Their ...
2 votes
2 answers
70 views

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

I've written my own SiteSearch Script in PHP. The SiteSearch parses two GET parameters from the queryString: search // the search-phrase filters (optional) // indicating which parts of the site to ...
0 votes
1 answer
1k views

Website not properly indexed by Google search console says "'noindex' detected in 'robots' meta tag" while this is not the case

TLTR I have a website (SPA) where SEO is very important. It was very poorly indexed and search console tells me it is due to 'noindex' in the robots meta, while I cannot see how Google comes to that ...
2 votes
1 answer
79 views

robots meta tag for specific web crawler

Web crawler list What I want noimageindex (Indexation-controlling parameters) will only work for googlebot (2 : two meta tag - code below) Others all crawlers (example as a YandexBot, Baiduspider) ...
2 votes
1 answer
180 views

What is a difference in the order of "follow" and "index" in a robots meta tag?

What is the difference between <meta name="robots" content="follow, index"> and <meta name="robots" content="index, follow"> Would either create ...
1 vote
0 answers
24 views

Can you specify the max-snippet length in a X-robots-tag header?

The specs by Google does not address this issue. So, I would like to know what is the x-robots-tag equivalent for: <meta name="googlebot" content="max-snippet:20"/> Is it ...
0 votes
1 answer
25 views

Ask How To Seting httaccses meta desc title same with oldies web

How do I setup .htaccses to get meta desc and title same with old web but redirect into new website example. aaaaaaa.blogspot.com = the old web - We want to get the meta desc and title from original ...
2 votes
1 answer
292 views

What are the potential side effects of the "nosnippet" robots meta tag?

Just recently I noticed our Search Console CTR drop from 4% to 2%. I noticed that we had an ever increasing number of impressions, but despite the increase in impressions, our clicks had decreased. ...
0 votes
2 answers
1k views

Disallowing URL parameter in robots.txt

I have important URLs on my site to which I’d like to append temporary parameters. For example: example.com/tool/input will have the parameter example.com/tool/input?foo input in the URL is ...
3 votes
2 answers
6k views

Robots meta tag with "noimageindex"

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
3 votes
1 answer
1k views

Can the content attribute of <meta name="robots" /> be left empty?

Is <meta name="robots" content="" /> a legitimate meta, exactly equivalent to: <meta name="robots" content="index, follow" /> <meta name="robots" content="all" /> [nothing] Or ...
4 votes
1 answer
228 views

Are there only 12 directives for <meta name="robots" />?

As far as I can tell, in 2017, the <meta name="robots" content="index, follow"> accepts the following directives in the content attribute: index follow all (equivalent to content="index, follow"...
2 votes
2 answers
42 views

Method for removal of high number of spammy links

I haven't been able to figure out the right combination for deindexing spammy links from search results. On one hand, we have the Google URL remover which removes links temporarily (they may show up ...
5 votes
2 answers
2k views

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

I want to have a chunk of JS in the middle of my page insert a meta-robots element into the <head> based on some arbitrary condition. Will Google and other search engines recognize this tag the ...
1 vote
3 answers
1k views

does google crawl/follow link on a page with meta robots noindex

We have added following meta tag on some of our pages: <meta content="noindex" name="robots"/> Does google crawl the content and links on this page and just don't show it on the google search ...
1 vote
1 answer
2k views

Use robots.txt to prevent privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google

I need to block pages such as privacy policy, terms and conditions, and guarantees from being crawled and indexed by Google. They might be unoriginal. I use them for my multiple websites and they ...
2 votes
1 answer
459 views

Help with robots meta tag and X-Robots-Tag

I have two questions about robots meta tag and X-Robots-Tag: If a page has both X-Robots-Tag HTTP header and robots meta tag, can this cause problems? For example: On HTTP response headers: X-...
0 votes
0 answers
25 views

can i deeplink my product pages which are not allowed to be crawled by robots

I have a set of pages which i have blocked by the robots and i don't want them to appear in search results. But if i am targetting users on this page via email/ad campaigns i want a possibility of ...
2 votes
1 answer
197 views

Meta tag robots noodp

Does Google still crawl a page when this is used? <meta name="robots" content="noodp"/> And when should this be used? Our robots.txt is set to index all.
4 votes
1 answer
158 views

URL in sitemap with temporary/dynamic noindex

Our scenario is we've got multiple pages set up for category-based browsing, such as example.com/products/boardgames A list of the canonical URLs for the categories is in the sitemap, with a frequency ...
5 votes
1 answer
12k views

What does "all" mean in meta robots tag?

Several websites have the following meta tag aimed at search engines like Google, Bing, etc.: <meta name="robots" content="all" /> What does it do?
1 vote
2 answers
1k views

difference between `<meta name="robots" content="nofollow" />` and `rel="nofollow"`

Is the meta tag <meta name="robots" content="nofollow" /> The same as rel="nofollow" in a link (apart from the fact the the meta tag would be page wide rather than on a per link basis) Or ...
1 vote
2 answers
325 views

Robots Meta Tag and query strings

I am trying to find out something about this tag: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> If I have a dynamic page like these example.com/product.aspx?ProductID=123 example.com/...
1 vote
1 answer
315 views

What are the reasons for using the meta tag "noindex, follow"?

Besides using the meta tag for hiding, say, the sitemap to prevent your site appearing spammy, are there any other situations for using noindex, follow?
2 votes
3 answers
516 views

Should I use both Robots.txt and META NAME="ROBOTS"?

I am currently using the Robots.txt file (the standard one of Joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior. Is there a reason to ...
5 votes
1 answer
1k views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
9 votes
2 answers
1k views

Which meta "robots" tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
1 vote
1 answer
56 views

meta robots for logged in vs. not logged in users

On a site I'm working on, The homepage has generic content for visitors who aren't logged in. When a user is logged in, the content shown is personalized to them and could even be considered private. ...
1 vote
1 answer
480 views

AdSense is adding a meta robots tag with noindex,nofollow - would this negatively affect my SEO?

When I add AdSense code to my webpage, I noticed that AdSense has a meta robots tag like below: <meta content="noindex,noarchive,nofollow" name="robots"> My page has no other meta robots tags. ...