Hot answers tagged

13

There are 2 main ways to prevent search engines from indexing specific pages: A Robots.txt file for your domain. The Meta Robots tag on each page. Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at ...


8

In the root of the directory of the subdomain website, add a file called robots.txt containing: User-agent: * Disallow: / This will tell web crawlers not to index the site at all. They do not have to obey, but the main ones will.


8

(I work with Google's crawling & indexing teams) Let's start with the easy one: assuming you don't want these files used for indexing at all, you can just use the robots.txt file to disallow crawling of the scripts and responses. This prevents Googlebot from accessing the URLs, and would of course also prevent it from using them for anything else. The ...


8

No, it does not influence in ranking, because that pages is not indexed at all, also it does not harmful for your site in some ways, but if you are placing too many noindex tags, then those pages will kept some PageRank or JuicyRank. Most of webmaster including me using noindex tag on specific directory, that have no quality content for example, list of ...


7

Yes. Eric Enge: Can a NoIndex page accumulate PageRank? Matt Cutts: A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page. Eric Enge: So, it can accumulate and pass PageRank. Matt Cutts: Right, and it will still accumulate PageRank, but it won't be showing in our Index. So, I ...


6

Here is what Google's John Mueller says about Noindex: in robots.txt: We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all. deepcrawl.com has done some testing of the feature and discovered that: It still works ...


5

You should prevent Google from crawling site search pages. Google doesn't want to crawl your site search at all. Here is Google's Matt Cutts blog post about the issue: Search results in search results by Matt Cutts on March 10, 2007. Google now actively penalizes sites that allow their site search results to be crawled and appear in Google's SERPs. By ...


5

There are various theories as to how Google knows what to crawl. It could be that someone linked to your mobile version. It could be that Google tried random urls and came across the /m version of your site. I'm not aware that they say they won't use URLs from their analytics data. Yes they do follow those rules: ...


4

No. Read this(all of it; there's a lot of useful stuff), though of particular relevance here: Google no longer recommends blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods. [...] Duplicate content on a site is not grounds for action on that site unless it appears that the intent ...


4

It sounds like your implementation is quite flawed. You should research best practice methods for serving mobile optimized content (using device detection - not cloaking, canonical link element) etc, rather trying to band-aid your current situation. Try looking at: https://www.google.com/support/webmasters/bin/answer.py?answer=35312 ...


4

If you mean this post, Google found it just fine when I searched for it. Considering you only posted it today, that's pretty good. I would've recommended using XML sitemaps and the HTTP ping feature to minimize indexing delays, but it looks like you're already using a plugin that does that. To be honest, I can't think of anything else to suggest — ...


4

Yoast answers this very well in this blog post: A better solution would be to add a <meta name="robots" content="noindex, follow"> tag to those search results pages, as it would prevent the search results from rankings but would allow the link “juice” to flow through to the returned posts and pages. Someone will inevitably link to a page you wish ...


4

Yoast discusses why this setting exists on his site. If your archive pages have any kind of static content or introduction, you run the risk of that content getting indexed on the second and subsequent pages of your archive and a dupe penalty applied. If you don't have that intro text, users get dumped into your older posts and may not have any real idea ...


4

From https://support.google.com/webmasters/answer/156184?hl=en Sitemaps are a way to tell Google about pages on your site we might not otherwise discover. This means you don't need a sitemap if you don't want Google to discover anything. Having a sitemap won't do anything so I wouldn't make one at all.


4

You should not use both noindex and rel=canonical, because noindex won't let pages pass any PageRank to its canonical version. rel="canonical" is a hint, not a directive, so Google will decide which page to take and show in search results. Here are posts at seroundtable, there they cite John Mueller's comments, from Google: No, you should not ...


3

Good catch! If you're blocking a page with robots.txt then crawlers will not able able to read the noindex meta tag. In these cases you should send the x-robots-tag HTTP header either via server-side code or .htaccess. .htaccess example: X-Robots-Tag: noindex Sample PHP code: header('X-Robots-Tag: noindex');


3

It is very hard to get local directory sites ranked in search engines these days regardless of whether or not the content is temporary. See this parody letter purportedly from Google, but really written by somebody unhappy that Google is not including directories in the search engine often. As for your temporary premium listings, I wouldn't change the URL ...


3

According to Google's Webmaster Guidelines: Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines. More often than not, these pages can come across as "spammy" and of low value.


3

A canonical link element is an HTML element that helps webmasters prevent duplicate content issues by specifying the "canonical" or "preferred" version of a web page. By default there can be numbers of different versions available for single page of your website. For example. http://www.example.com/ http://www.example.com/index.html http:/example.com/ ...


3

Days, Weeks and Months... It can take Google days, weeks and even months for Google to remove pages marked noindex, robots and 404's. Generally it takes Google several crawls before Google acts on the new information of a page. User Errors It should be also noted that more than often users make human errors and create 404's, noindexs and so forth by ...


3

The noindex should be enough to stop it getting indexed. With regards to iframes and linking, I can't actually find any guidelines from Google. In fact, all they seem to officially say on the entire subject of iframes is "Google supports frames and iframes to the extent that it can." You would need to link to the article you mention for its context, but I ...


3

TLDR; No So, for example, if I send noindex for my jQuery script, so Google would not be able to use them to load Ajax, I suppose it is not good for my site's SEO, right? No, you seem confused what NOINDEX actually does. NOINDEX: Allows crawling, following the links in it. Disallows indexing (which would automatically include NOARCHIVE and ...


3

It makes a difference. The bots do this: Go to noindex.html. Crawl it, check it, do all the usual checks, get all internal links Don't index this page, because rel="noindex" Repeat for each internal link As you can see, it still follows all internal links, it just doesnt index no-index.html. This means internal-linked-page.html will get found, crawled ...


3

First things first. Google does not index certain file types such as CSS, JavaScript, images, and so on. But, you say, Google does index images. Well yes and no. It indexes images by using the text surrounding the image and the link values associated with the image. Yes, during growing pains while updating their capabilities, Google has indexed some of these ...


3

To prevent Google to index your documents but let bots crawl your documents (for SEO purposes), you can put these following lines in your .htaccess (if you use Apache as a web server). It will give weight to tell to Google to index your webpages and not your documents. <Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex" </Files> With ...


3

Perhaps a very non-user friendly site-map?


3

The surest way to get those pages out of index is to use this in the HTML head section: <meta name="robots" content="noindex" /> Sometimes Google indexes some URLs (but not the content) despite the Disallow in robots.txt. See my answer to this question for details.


3

The problem is that you are using noindex on your pages, see line:97 when viewing the source of your page. See below: Line 96: <meta name="description" content="xxxx" /> Line 97: <meta name="robots" content="noindex,nofollow" /> Line 98: <meta name="googlebot" content="noarchive" />


3

Using a robots.txt file in your subdomain will help (and Google will obey this), but another step you can take is to specify with a Google Webmasters account that you don't want this subdomain to be indexed. You can also use a meta tag on all pages in the subdomain: <meta name="robots" content="noindex"> If this happens to be a site that you ...


3

It's important to note that nofollow, noindex and even blocking via robots doesn't necessary mean that the content won't be crawled, in fact these pages can still be indexed but rather hidden from public search results (Yes Google is naughty, but it true). You see when using noindex on the page Google needs to crawl the page to find that tag out, Googlebot ...



Only top voted, non community-wiki answers of a minimum length are eligible