Hot answers tagged noindex
4
There are various theories as to how Google knows what to crawl. It could be that someone linked to your mobile version. It could be that Google tried random urls and came across the /m version of your site. I'm not aware that they say they won't use URLs from their analytics data.
Yes they do follow those rules: ...
4
No. Read this(all of it; there's a lot of useful stuff), though of particular relevance here:
Google no longer recommends blocking
crawler access to duplicate content on
your website, whether with a
robots.txt file or other methods.
[...]
Duplicate content on a site is not
grounds for action on that site unless
it appears that the intent ...
4
If you mean this post, Google found it just fine when I searched for it. Considering you only posted it today, that's pretty good.
I would've recommended using XML sitemaps and the HTTP ping feature to minimize indexing delays, but it looks like you're already using a plugin that does that. To be honest, I can't think of anything else to suggest — ...
3
You only really have one page in the site and that page doesn't have a great deal of content on it. I appreciate there is an about page, but that isn't coded properly and has no meta description so google is ignoring it.
My advice would be:-
Write more content (game tutorials, a blog, high score pages and so on)
Make sure your HTML is well formed and SEO ...
3
Using a robots.txt file in your subdomain will help (and Google will obey this), but another step you can take is to specify with a Google Webmasters account that you don't want this subdomain to be indexed. You can also use a meta tag on all pages in the subdomain:
<meta name="robots" content="noindex">
If this happens to be a site that you ...
2
because I want to know precisely which pages my visitors are intereted in.
Your visitors are interested in the pages that come up highest in your stats. (I'm necessarily simplifying here, but generally.) Finding that out doesn't require disappearing everything you assume is not interesting.
Let's say for the sake of discussion that your tag page for ...
2
If you use the X-Robots-Tag tag, either as a meta tag or HTTP header with a value of noindex then that page will not appear in the SERPs. The links should be followed unless you use nofollow in the links, X-Robots-Tag meta tag, or X-Robots-Tag HTTP header on the HTML sitemap page.
2
You can set the rate at which Googlebot crawls your website in Google Webmaster Tools.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620
This is how you should limit Googlebots access not by denying it access. Because noindex pages they will still visit causing hits on your server. Robots.txt disallow may work but be careful you ...
2
If you have no descriptive text for your products you will have lots of "thin content" pages that will not be viewed favorably by search engines such as Google.
On the other hand, if you have repeated descriptions you will have lots of "duplicate content" that will not be viewed favorably, either.
Unfortuantely, there simply is no automated solution to ...
2
Or should I use rel="canonical" for the promotion page?
No, only use canonical for pages with (almost) identical content. Your promotion list page and an individual promotion are clearly different pages.
What I do is use "noindex, follow" for the links.
Is that a good idea?
I think you shouldn't disallow the indexing. Search engines can be very ...
2
It's important to note that nofollow, noindex and even blocking via robots doesn't necessary mean that the content won't be crawled, in fact these pages can still be indexed but rather hidden from public search results (Yes Google is naughty, but it true). You see when using noindex on the page Google needs to crawl the page to find that tag out, Googlebot ...
2
This problem often occurs with Wordpress websites.
You get "not found pages" (404) because Googlebot find some links on the source code of your website. Wordpress in particular add some links for feeds in the <head> section even if you don't want. You can see these links by displaying the source code of your webpages (CTRL + U with Google Chrome, ...
2
The problem is that you are using noindex on your pages, see line:97 when viewing the source of your page. See below:
Line 96: <meta name="description" content="xxxx" />
Line 97: <meta name="robots" content="noindex,nofollow" />
Line 98: <meta name="googlebot" content="noarchive" />
2
Solutions from Kenzo and Paul are good, you can put meta tags noindex on your web pages and add robots.txt to disallow robots.
But in my opinion, the best solution is to use password authentification on your sub domain. This is the only solution you're sure robots can access and index your web site. If you use Apache, you can implement htpasswd.
1
If you know the content is low quality then blocking it isn't a bad idea. That will make you less susceptible to potential Pandaenalties. But only do that if you're sure that content is low quality. Sometimes comments contain useful information that someone may be looking for and by blocking it you're missing out on targetted traffic. Pus Google is good at ...
1
No, it wont help. A penalty is not applied to a single page, but rather applied to the entire domain name.
If you want to stick to Google's good pages, I suggest you try removing all the spam links and then report it to Google through the Webmasters Tools. Even if you have not removed all the links, Google shall reward you for your efforts by getting your ...
1
Google is pretty vague on how the penguin penalty works from spammy links with keywords so your never an official answer, in fact Google doesn't want people knowing otherwise blackhatters would reverse engineer every-time they got hit.
As far as I've learned from clients that I've worked with is that penalties are not just applied to the page that is being ...
1
That looks like it should be correct, per instructions from Google. A few suggestions:
Make sure the meta tags are within the <head> tag
Make sure the meta tags actually say "noindex, nofollow" (your code says "noindex, follow" - not sure if that's just a copying mistake)
Use the standard <meta name="robots" content="noindex, nofollow"> instead ...
1
The problem you face mostly is the fact that the demos could be considered thin content and ultimately effect your website rankings. You could use no-index on theses but even then some juice gets leaked even when you use no-follow and this problem still occurs if you was to host the content on a sub domain.
For ranking purposes this is what I propose:
...
1
...currently showing in the SEO results?
The other answers are more about proactively preventing the indexing of a (sub)domain (which is primarily what you are asking in the question) rather than actively removing your pages from the search results, which might be more what you are after, judging by your other question.
You still need to block your ...
1
The fact that a page is being served up with a 404 error means that responsible web indexes will not be storing the 404 page itself anyway. That is what the HTTP status code is for - the number 404 isn't there for the user's benefit! As far as removing the original page from an index, it would be better to make resources which are truly gone return an ...
1
Yes you can do this by setting the priority within the sitemap.xml
see http://en.wikipedia.org/wiki/Sitemaps#Element_definitions
example
<url>
<loc>http://www.foo.com</loc>
<lastmod>2011-06-03</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority></url>
</url>
So in your case ...
1
My first answer is: why would you want less exposure on Google?
Surely people will still get to the article, even if they have to go via the listing page. And that is a good thing.
Maybe rather than stopping Google from indexing certain pages, why not do some work on the articles to get them higher. Look at the text on the pages as well as the keywords, ...
Only top voted, non community-wiki answers of a minimum length are eligible
