Tag Info

Hot answers tagged

13

There are 2 main ways to prevent search engines from indexing specific pages: A Robots.txt file for your domain. The Meta Robots tag on each page. Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at ...


5

There are various theories as to how Google knows what to crawl. It could be that someone linked to your mobile version. It could be that Google tried random urls and came across the /m version of your site. I'm not aware that they say they won't use URLs from their analytics data. Yes they do follow those rules: ...


4

No. Read this(all of it; there's a lot of useful stuff), though of particular relevance here: Google no longer recommends blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods. [...] Duplicate content on a site is not grounds for action on that site unless it appears that the intent ...


4

If you mean this post, Google found it just fine when I searched for it. Considering you only posted it today, that's pretty good. I would've recommended using XML sitemaps and the HTTP ping feature to minimize indexing delays, but it looks like you're already using a plugin that does that. To be honest, I can't think of anything else to suggest — ...


4

Yoast answers this very well in this blog post: A better solution would be to add a <meta name="robots" content="noindex, follow"> tag to those search results pages, as it would prevent the search results from rankings but would allow the link “juice” to flow through to the returned posts and pages. Someone will inevitably link to a page you wish ...


4

From https://support.google.com/webmasters/answer/156184?hl=en Sitemaps are a way to tell Google about pages on your site we might not otherwise discover. This means you don't need a sitemap if you don't want Google to discover anything. Having a sitemap won't do anything so I wouldn't make one at all.


3

The surest way to get those pages out of index is to use this in the HTML head section: <meta name="robots" content="noindex" /> Sometimes Google indexes some URLs (but not the content) despite the Disallow in robots.txt. See my answer to this question for details.


3

The problem is that you are using noindex on your pages, see line:97 when viewing the source of your page. See below: Line 96: <meta name="description" content="xxxx" /> Line 97: <meta name="robots" content="noindex,nofollow" /> Line 98: <meta name="googlebot" content="noarchive" />


3

Using a robots.txt file in your subdomain will help (and Google will obey this), but another step you can take is to specify with a Google Webmasters account that you don't want this subdomain to be indexed. You can also use a meta tag on all pages in the subdomain: <meta name="robots" content="noindex"> If this happens to be a site that you ...


3

You only really have one page in the site and that page doesn't have a great deal of content on it. I appreciate there is an about page, but that isn't coded properly and has no meta description so google is ignoring it. My advice would be:- Write more content (game tutorials, a blog, high score pages and so on) Make sure your HTML is well formed and SEO ...


3

Yoast discusses why this setting exists on his site. If your archive pages have any kind of static content or introduction, you run the risk of that content getting indexed on the second and subsequent pages of your archive and a dupe penalty applied. If you don't have that intro text, users get dumped into your older posts and may not have any real idea ...


3

To prevent Google to index your documents but let bots crawl your documents (for SEO purposes), you can put these following lines in your .htaccess (if you use Apache as a web server). It will give weight to tell to Google to index your webpages and not your documents. <Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex" </Files> With ...


3

You should prevent Google from crawling site search pages. Google doesn't want to crawl your site search at all. Here is Google's Matt Cutts blog post about the issue: Search results in search results by Matt Cutts on March 10, 2007. Google now actively penalizes sites that allow their site search results to be crawled and appear in Google's SERPs. By ...


3

Days, Weeks and Months... It can take Google days, weeks and even months for Google to remove pages marked noindex, robots and 404's. Generally it takes Google several crawls before Google acts on the new information of a page. User Errors It should be also noted that more than often users make human errors and create 404's, noindexs and so forth by ...


2

If multiple page URLs produce the same content then that's exactly what <link rel="canonical"> is for. It's for telling the search engines that the content of multiple URLs are the same and to use that specific one as the primary one. That avoids duplication issues altogether and is very simple to do.


2

because I want to know precisely which pages my visitors are intereted in. Your visitors are interested in the pages that come up highest in your stats. (I'm necessarily simplifying here, but generally.) Finding that out doesn't require disappearing everything you assume is not interesting. Let's say for the sake of discussion that your tag page for ...


2

If you use the X-Robots-Tag tag, either as a meta tag or HTTP header with a value of noindex then that page will not appear in the SERPs. The links should be followed unless you use nofollow in the links, X-Robots-Tag meta tag, or X-Robots-Tag HTTP header on the HTML sitemap page.


2

Or should I use rel="canonical" for the promotion page? No, only use canonical for pages with (almost) identical content. Your promotion list page and an individual promotion are clearly different pages. What I do is use "noindex, follow" for the links. Is that a good idea? I think you shouldn't disallow the indexing. Search engines can be very ...


2

The fact that a page is being served up with a 404 error means that responsible web indexes will not be storing the 404 page itself anyway. That is what the HTTP status code is for - the number 404 isn't there for the user's benefit! As far as removing the original page from an index, it would be better to make resources which are truly gone return an ...


2

You can set the rate at which Googlebot crawls your website in Google Webmaster Tools. http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620 This is how you should limit Googlebots access not by denying it access. Because noindex pages they will still visit causing hits on your server. Robots.txt disallow may work but be careful you ...


2

If you have no descriptive text for your products you will have lots of "thin content" pages that will not be viewed favorably by search engines such as Google. On the other hand, if you have repeated descriptions you will have lots of "duplicate content" that will not be viewed favorably, either. Unfortuantely, there simply is no automated solution to ...


2

Solutions from Kenzo and Paul are good, you can put meta tags noindex on your web pages and add robots.txt to disallow robots. But in my opinion, the best solution is to use password authentification on your sub domain. This is the only solution you're sure robots can access and index your web site. If you use Apache, you can implement htpasswd.


2

It's important to note that nofollow, noindex and even blocking via robots doesn't necessary mean that the content won't be crawled, in fact these pages can still be indexed but rather hidden from public search results (Yes Google is naughty, but it true). You see when using noindex on the page Google needs to crawl the page to find that tag out, Googlebot ...


2

This problem often occurs with Wordpress websites. You get "not found pages" (404) because Googlebot find some links on the source code of your website. Wordpress in particular add some links for feeds in the <head> section even if you don't want. You can see these links by displaying the source code of your webpages (CTRL + U with Google Chrome, ...


2

Search engines might … … find your testing website. … index pages from your testing website. … decide that a page from the real site and the testing site are duplicate copies, so they might only index one of those pages. If you are unlucky, they might decide to index the testing page instead of the real page in some cases (especially, if not all of your ...


2

If you marked those pages with a noindex meta tag, there are two possible reasons I see: Googlebot didn't reindex the site yet and you don't have wait enough (it can take a lot of time like several weeks, it depends on the site crawl rate) You use 404 instead of 410 HTTP status (Gone)


2

You should be fine with noindexing the page, as there is no legal requirement to have T&C pages indexed (nor even included on a site). However you might want to do a back link check and check traffic to the page in GA first; perhaps people are linking to the page (unlikely) or it gets a lot of traffic and this could be why it keeps getting picked? If ...


1

No, it wont help. A penalty is not applied to a single page, but rather applied to the entire domain name. If you want to stick to Google's good pages, I suggest you try removing all the spam links and then report it to Google through the Webmasters Tools. Even if you have not removed all the links, Google shall reward you for your efforts by getting your ...


1

Google is pretty vague on how the penguin penalty works from spammy links with keywords so your never an official answer, in fact Google doesn't want people knowing otherwise blackhatters would reverse engineer every-time they got hit. As far as I've learned from clients that I've worked with is that penalties are not just applied to the page that is being ...


1

You could submit an XML sitemap of the OLD site URLs trough Webmaster Tools, if you still have information about those old URLs. This way Google would both recognize the redirects and find the new URLs as well. I did this for a client once, to speed up things when the site was redesigned. You could also share some of the new URLs on Google+, as this often ...



Only top voted, non community-wiki answers of a minimum length are eligible