2

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing.

The site was launched in April 2015. The total number of pages on the site tends to infinity because the content is generated dynamically and based on parameters selected by the end user.

I've selected 128,000 pages for indexing; these pages are listed in sitemaps. All other pages contain the meta tag <meta name="robots" content="noindex"/>, and all links to these pages are tagged with rel="nofollow".

But during the first two months, these tags (noindex, rel="nofollow") had not been included in the pages. Today, the total number of indexed pages exceeds 4 million.

Googlebot continues to read unwanted pages, and according to server stats it seems that Googlebot ignores "nofollow" attributes. Why does it happen? What should be done to stop it reading and indexing these unwanted pages?

The total number of indexed pages is still increasing.

How can I request a re-scan of the previously indexed pages (so that Googlebot can re-read <meta name="robots" content="noindex" />)?

Example of pages that should be indexed: http://example.com/en/indicators/GDP_current_prices/Philippines

Pages, that should not be indexed:
http://example.com/en/indicators/GDP_current_prices/India-Philippines/

  • 2
    AFAIK Google does not follow nofollow links. However, nofollow does not prevent pages from being found (by other means) and indexed, it primarily prevents the link from being used as a ranking signal (when linking to untrusted content). The important value here is noindex, as mentioned. The real question is... "How to get these pages removed as quick as possible". The noindex tag will work, but it will take time, as Google needs to recrawl the page. – MrWhite Jul 2 '15 at 16:49
  • @wd3 Sounds like an answer to me. The links should contain nofollow and noindex. However, whey the noindex on each page was not picked up would have to be investigated. If this is hard-coded into the page, that should work. If this is placed into the page via some mechanism, then I suggest making sure this works as expected. Google is very good at obeying the rules they helped to lay out. – closetnoc Jul 3 '15 at 0:07
  • In Google search console (webmaster tools) you can do a URL removal for entire subfolders. If all the pages you do not wanted indexed reside in the same subfolder/s and there aren't' pages within there that you do want indexded, you can request the subfolder be removed using the URL removal tool. As long as all the pages have the noindex tag they will be removed from Google a lot faster than waiting for Google to honour the noindex tag. – Max Jul 3 '15 at 2:13
1

rel="nofollow" is not meant to prevent indexing of the linked page as that is the way that Google locates new pages on the internet. All it does is tell Google not to pass link juice to that linked page in an attempt to mitigate link spamming. The only way to block those old pages from Google would be to add each one to your robots.txt file as a disallowed page then give it a few weeks until Google reindexes your whole site and sees the robots.txt file. Once it see's that it will remove the pages from the index. The only other way is to manually remove each and every page one at a time from the index using a web form but that only keeps them off for i believe 2 months then they can be reindexed again and added to the Google index for searching again. robots.txt is the standard for excluding indexing of certain pages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.