Hot answers tagged

9

Yes. Eric Enge: Can a NoIndex page accumulate PageRank? Matt Cutts: A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page. Eric Enge: So, it can accumulate and pass PageRank. Matt Cutts: Right, and it will still accumulate PageRank, but it won't be showing in our Index. So, I wouldn't ...


9

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work. Here is what Google's John Mueller says about Noindex: in robots.txt: We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don'...


9

Yes, Google still crawl webpages that have noindex tag. But if you have same content on two different webpages and one URL contain noindex tag, while second does not, then you should not worry about it, because out of all duplicate content only one webpage is indexed by Google. Rest of webpages are crawlable but not indexed in Google search result, so that ...


8

(I work with Google's crawling & indexing teams) Let's start with the easy one: assuming you don't want these files used for indexing at all, you can just use the robots.txt file to disallow crawling of the scripts and responses. This prevents Googlebot from accessing the URLs, and would of course also prevent it from using them for anything else. The ...


8

No, it does not influence in ranking, because that pages is not indexed at all, also it does not harmful for your site in some ways, but if you are placing too many noindex tags, then those pages will kept some PageRank or JuicyRank. Most of webmaster including me using noindex tag on specific directory, that have no quality content for example, list of ...


8

There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed. If this was a PHP page then you could obviously just send a 404 Not Found header as part of the standard response - to make sure that it always returns a 404. If you are on the Apache web server then you could also use .htaccess (mod_rewrite) to force any ...


8

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like: X-Robots-Tag: noindex After implementing the HTTP header, remove the disallow from your robots.txt. ...


7

You should prevent Google from crawling site search pages. Google doesn't want to crawl your site search at all. Here is Google's Matt Cutts blog post about the issue: Search results in search results by Matt Cutts on March 10, 2007. Google now actively penalizes sites that allow their site search results to be crawled and appear in Google's SERPs. By ...


7

As Goyllo has already stated, search engine bots will crawl pages that have a noindex meta tag. If you think about it, they need to crawl the page in order to see the noindex meta tag in the first place. (You could use an X-Robots-Tag HTTP response header instead and, in theory, a bot would only need to do a HEAD request in order to see the noindex attribute ...


7

"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google. A NOINDEX tag looks like the below and it should be placed in the section of any page you do not want indexed: <meta name="robots" content="noindex"> More information can be found here. In ...


7

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it! Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if ...


6

Note that Noindex is not part of the original robots.txt specification. Google supported it as experimental feature (see: How does “Noindex:” in robots.txt work?), but it’s not clear if that is still the case (as they didn’t document it to begin with). But let’s assume it is. Your robots.txt has two problems. Empty lines A record must not contain empty ...


5

First things first. Google does not index certain file types such as CSS, JavaScript, images, and so on. But, you say, Google does index images. Well yes and no. It indexes images by using the text surrounding the image and the link values associated with the image. Yes, during growing pains while updating their capabilities, Google has indexed some of these ...


5

Nick's answer to add robots.txt might not stop Google from index, it just says stop crawling but Google might index - official link here. Adding meta noindex is too risky if you are working with a team that is not aware of implications of noindex tag, if it is pushed to production env. <meta name="robots" content="noindex"> I would recommend you ...


4

From https://support.google.com/webmasters/answer/156184?hl=en Sitemaps are a way to tell Google about pages on your site we might not otherwise discover. This means you don't need a sitemap if you don't want Google to discover anything. Having a sitemap won't do anything so I wouldn't make one at all.


4

You should not use both noindex and rel=canonical, because noindex won't let pages pass any PageRank to its canonical version. rel="canonical" is a hint, not a directive, so Google will decide which page to take and show in search results. Here are posts at seroundtable, there they cite John Mueller's comments, from Google: No, you should not ...


4

Good catch! If you're blocking a page with robots.txt then crawlers will not able able to read the noindex meta tag. In these cases you should send the x-robots-tag HTTP header either via server-side code or .htaccess. .htaccess example: X-Robots-Tag: noindex Sample PHP code: header('X-Robots-Tag: noindex');


4

A canonical link element is an HTML element that helps webmasters prevent duplicate content issues by specifying the "canonical" or "preferred" version of a web page. By default there can be numbers of different versions available for single page of your website. For example. http://www.example.com/ http://www.example.com/index.html http:/example.com/ http:...


4

I am not sure if your theory is good because it is based on a very big assumption that your new website will take the credit for your old site after deindexing. Except rel="canonical", you don't have anything else to pass on the credit and if that isn't working now, how can you be sure it will work after deindexing your old pages? Understand that rel="...


4

You only really need to set the rel="canonical" header. This should be sufficient in ensuring only the canonical URL (ie. the one with no URL params) appears in the SERPs. Setting a noindex robots meta tag for such URLs would seem to be overkill (and a tad risky) IMO. Presumably you are unable to set a rel="canonical" meta tag in the HTML itself? ...what ...


4

You can take either path here. What matters is how you go about it. Let's say you decide to delete those articles. That should be fine. Just make sure you 301 redirect those URL's to something better that's hopefully contextually similar. Take those articles out of the XML sitemap, too, and resubmit the sitemap. Before doing this, it may be worth it to check ...


4

You are sending Google two conflicting commands: you are telling them to NOINDEX some pages but then you are also preventing them from reading the NOINDEX header by blocking them from crawling these pages (in the robots.txt) so they can't see the NOINDEX command. You should only use the NOINDEX tag and let their bots crawl your website in order to see it; ...


3

Google recently (December 2017) said that noindex,follow pages end up not not passing link juice. Initially link juice will get passed, but eventually the links on the noindex page will be ignored. Long term, noindex,follow is equivelent to noindex,nofollow. See: Google: Long Term Noindex Will Lead To Nofollow Also


3

No, the NINDEX, NOFOLLOW on the embedded iframe will not effect the page it is displaying on. As they are neither on the actual page, nor within the <head> section of the pages source code. I know this from experience as I have used this technique before where I had content that must appear on the page, but was duplicate content, so I didn't want ...


3

The noindex should be enough to stop it getting indexed. With regards to iframes and linking, I can't actually find any guidelines from Google. In fact, all they seem to officially say on the entire subject of iframes is "Google supports frames and iframes to the extent that it can." You would need to link to the article you mention for its context, but I ...


3

It makes a difference. The bots do this: Go to noindex.html. Crawl it, check it, do all the usual checks, get all internal links Don't index this page, because rel="noindex" Repeat for each internal link As you can see, it still follows all internal links, it just doesnt index no-index.html. This means internal-linked-page.html will get found, crawled and ...


3

Days, Weeks and Months... It can take Google days, weeks and even months for Google to remove pages marked noindex, robots and 404's. Generally it takes Google several crawls before Google acts on the new information of a page. User Errors It should be also noted that more than often users make human errors and create 404's, noindexs and so forth by error,...


3

If you are redirecting old pages to the new pages that have the same content as the redirected pages (and ideally the same URL structure), then patience is needed. I suggest that you look at the analytics of the old domains and see which pages brought most of the traffic. Track those to see whether or not you regained the old traffic. But, if you are doing ...


3

According to Google's Webmaster Guidelines: Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines. More often than not, these pages can come across as "spammy" and of low value.


Only top voted, non community-wiki answers of a minimum length are eligible