Hot answers tagged

9 votes
Accepted

How does "Noindex:" in robots.txt work?

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work. Here is what Google's John Mueller says about Noindex: ...
9 votes

Do search engines still crawl a noindex page

Yes, Google still crawl webpages that have noindex tag. But if you have same content on two different webpages and one URL contain noindex tag, while second does not, then you should not worry about ...
  • 5,497
8 votes
Accepted

Does the Title and meta description for noindex pages influence SEO ranking in any way for the rest of the indexed pages?

No, it does not influence in ranking, because that pages is not indexed at all, also it does not harmful for your site in some ways, but if you are placing too many noindex tags, then those pages will ...
  • 5,497
8 votes
Accepted

Ok to use meta noindex on 404 pages?

There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed. If this was a PHP page then you could obviously just send a 404 Not Found header as part of the ...
  • 42k
8 votes
Accepted

How do I "noindex" a text (.txt) file?

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. ...
8 votes
Accepted

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek ...
7 votes
Accepted

Do search engines still crawl a noindex page

As Goyllo has already stated, search engine bots will crawl pages that have a noindex meta tag. If you think about it, they need to crawl the page in order to see the noindex meta tag in the first ...
  • 42k
7 votes

URLs with 'NoIndex` in robots.txt are being indexed by Google

"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google. A NOINDEX tag looks like the below and ...
6 votes

X-Robots-Tag: noindex on an HTTP 302 response to remove subdomain

As of December 2022, Google will respect a X-Robots-Tag HTTP Response header that comes as part of a HTTP 302 response - Google Search Console will report: "Indexing allowed? No: 'noindex' ...
6 votes

URLs with 'NoIndex` in robots.txt are being indexed by Google

Note that Noindex is not part of the original robots.txt specification. Google supported it as experimental feature (see: How does “Noindex:” in robots.txt work?), but it’s not clear if that is still ...
  • 21.6k
5 votes

No index for an entire site

Nick's answer to add robots.txt might not stop Google from index, it just says stop crawling but Google might index - official link here. Adding meta noindex is too risky if you are working with a ...
  • 526
5 votes
Accepted

Link rel="nofollow" vs robots.txt disallow for "/search" URLs?

Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt. However, that doesn't necessarily prevent the /search pages from appearing as ...
  • 42k
4 votes
Accepted

if an HTTP page can be cached by proxy should I put "noindex" in the HTTPS version or not?

A canonical link element is an HTML element that helps webmasters prevent duplicate content issues by specifying the "canonical" or "preferred" version of a web page. By default there can be numbers ...
4 votes

Will meta setting Meta="NOINDEX" eventually remove page from Google index?

I am not sure if your theory is good because it is based on a very big assumption that your new website will take the credit for your old site after deindexing. Except rel="canonical", you don't have ...
  • 1,316
4 votes
Accepted

How to correctly remove MULTIPLE parameters from the Google index using htaccess and canonical / noindex?

You only really need to set the rel="canonical" header. This should be sufficient in ensuring only the canonical URL (ie. the one with no URL params) appears in the SERPs. Setting a noindex robots ...
  • 42k
4 votes
Accepted

Is there differences between deleting low quality articles vs no-indexing low quality articles SEO wise?

You can take either path here. What matters is how you go about it. Let's say you decide to delete those articles. That should be fine. Just make sure you 301 redirect those URL's to something better ...
4 votes

How to prevent google index for a specific subpath?

You are sending Google two conflicting commands: you are telling them to NOINDEX some pages but then you are also preventing them from reading the NOINDEX header by blocking them from crawling these ...
4 votes
Accepted

Deindex a site through Google Search Console without access to the site's server to make changes?

You Can Submit a Removal Request Submitting will remove URLs for 6 months, this will buy you some time until you're able to put a permanent solution in place. In your case, you might find the prefix ...
  • 5,731
3 votes

How do search engines behave when encountering noindex?

What I don't know is whether it also blocks the URL from showing up in search results, or just that particular content with that URL. If you have a noindex robots tag then the URL should not appear ...
  • 42k
3 votes

X-Robots-Tag: noindex on an HTTP 302 response to remove subdomain

Google considers the status code before looking at everything else, so the X-Robots-Tag will be ignored. We did some test a year ago with a client and this was not working. The strategy that we used ...
  • 181
3 votes

Do noindex,follow pages pass pagerank / link juice?

Google recently (December 2017) said that noindex,follow pages end up not not passing link juice. Initially link juice will get passed, but eventually the links on the noindex page will be ignored....
3 votes

Should I set NOINDEX header for my JS, CSS and image files?

Nowadays, don't worry about it. Although we don't set the x-robot http header noindex for static file css, js, Googlebot know what file should be indexed or not. Except image, because the images can ...
3 votes

How to stop certain urls from being indexed

If your goal is for this pages to not be seen by the public, it's best to put a password on this set of pages. And/or have some configuration that only allows specific, whitelisted addresses able to ...
3 votes

What are the reasons for using the meta tag "noindex, follow"?

Faceted navigation / search filtering allows visitors to narrow their search for specific products. Such navigation features generate crawl paths that are of no real value to a search engine but are ...
3 votes

Is it 100% safe to add NOINDEX tag to any duplicate/copy content issues?

Both rel=canonical and noindex has their own purposes. rel=canonical should be used when you want to give weightage of the page to a page from where the content originated. The seo juice in this case ...
  • 1,694
3 votes
Accepted

Is Google starting to ignore link directories such as brownbook

Generally speaking, Google does not like directory sites too much. Some perhaps. However, most are generally either low quality or spammy or even replicated information that can be found everywhere. ...
  • 32.7k
3 votes
Accepted

SEO put no-indexed pages in sitemap or not?

I'm kind of wondering if I've missunderstood your question, but... there is no point putting "noindex" pages in your XML sitemap. An XML sitemap advertises the pages you want indexed. So, including a ...
  • 42k
3 votes
Accepted

Will Google automatically re-index my page when I remove the 'noindex' tag

If you want to request indexing, you can submit it to Search Console. This tends to speed the process by manually indicating you'd like a page (and linked pages) considered for indexing. If you ...
  • 3,933
3 votes

How to use X-Robots-Tag to noindex all subdomains (save specific exceptions)?

Since all the subdomains point to the same place on the filesystem then you can do something like the following at the top of your .htaccess file in the document root: SetEnvIf Host ^docs\. INDEX ...
  • 42k
3 votes
Accepted

Google indexing Wordpress in subdirectory disallowed in robots.txt

Using Disallow in robots.txt doesn't prevent search engines from indexing pages, only from crawling them - this is a common SEO myth. To answer your questions: Googlebot finds pages not only through ...

Only top scored, non community-wiki answers of a minimum length are eligible