Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

On some of our sites, we now have several thousand pages that dilute our website's keyword density. The website is an MVC site with SEO routing.

If I submit a new sitemap with say only the 2000 or so pages that we want indexed, even though navigating to the diluting pages still works, will Google re-index the site with only those 2000 pages, dropping the superfluous ones?

For example, I want to keep roughly 2000 of the following:

www.mysite.com/some-search-term-1/some-good-keywords
www.mysite.com/some-search-term-2/some-more-good-keywords

And remove several thousand of the following that have already been indexed.

www.mysite.com/some-search-term-xx/some-poor-keywords
www.mysite.com/some-search-term-xx/some-poor-more-keywords

These pages are not actually "removed" as navigating to these URL's still renders a page. Even though there are potentially hundreds of thousands of pages, I only want say 2000 to be re-indexed and retained. The others removed (without having to do these manually).

Thanks.

share|improve this question
    
support.google.com/webmasters/bin/… ? –  j0k Oct 24 '12 at 21:47
    
Because the pages are routed using friendly URL's the page is not actually "removed" from the site. See update. –  ElHaix Oct 24 '12 at 22:17
add comment

3 Answers

up vote 0 down vote accepted

You can tell Google to not crawl specific parts of your site with robots.txt, you can use the sitemap, you can use a few other technics, but in the end, if there is a way yo get to the pages following links or threads from a forums, or pages from somebody else, Google will find it.

If those pages, the good and the bad ones are related enough, you can use the canonical meta to help redirect the traffic to those pages that you want, but they have to be similar, otherwise you will be using the tag badly.

If you have a forum, it will be much better to do some clean up and remove old, inaccurate or unfinished threads, those are not useful and dilute your ranking.

If you have other kind of site, you may provide a description to it or a link so we can tell you a better approach.

share|improve this answer
    
I created a dynamic site map. And submitted it through WebmasterTools. The issue is that the pages are generated from search terms. So if I add a noindex metatag to result pages that are not in the sitemap, would that do the trick? What about denying all links (/) in robots.txt and adding only the valid links (2000 or so) in there (kinda messy)? –  ElHaix Oct 25 '12 at 1:10
    
it is messy, but it may work, plus remember that robots.txt is something they may use, but they may ignore it and you can not control what the users will link in other places/sites, and Google may find and follow those links even if you don't want. But if the page you want to "hide" is completely disconnected to the site and only reachable via a search, then you only have to worry about users linking to those pages, which will be a small possibility and so, it will work, at least most of the times. –  PatomaS Oct 25 '12 at 1:26
    
Ok... so scrap the robots.txt idea, and I would like to not have to dynamically add noindex to the messy pages. All of the messy links have been removed (changed to Javascript links), good links sitemap submitted. I'll have to give it a week or so to see if there is any decline in indexed pages, but hope this works. So even though a URL is valid on a site, if there is no actual link to that page, will it be removed from the index (trying to avoid having to add the noindex)? –  ElHaix Oct 25 '12 at 13:09
add comment

An XML Sitemap doesn't govern the content that a search engine will index. It's an aid to discovery, but whether or not a piece of content is listed in a Sitemap has nothing to do with whether or not it's indexed. Info here.

Robots.txt, as discussed above, may prevent crawling but will not prevent indexing. If a page is blocked by robots.txt but is linked to, either internally or externally, it stands a chance of appearing in the index, with "blocked by robots.txt" notification in the result snippet.

I believe the only reliable way to do what you want is either noindex: either HTML meta tag or the X-Robots-Tag equivalent.

However, I question the whole premise. I wouldn't be concerned with "diluting your website's keyword density" - it's basically meaningless. Does the content have value and purpose? If so, are visitors finding it easily, or could it be improved? If the content is poor and serves no purpose, get rid of it. Base this process on - and assess your progress with - analytics data.

share|improve this answer
add comment

Another way is through Google Webmaster Tools on the left menu, under Optimization. I have done this before and is very quick (more than updating your robots.txt)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.