Upon first submitting our sitemap, we didn't take into account the index of archive pages. With 50,000 articles, we're seeing lots of archive pages results (i.e. News (Page 40 of 876))

Since then we've placed <meta name="robots" content="noindex"> in the head of our archives template and resubmitted the sitemap. Now, Google is indexing the sitemap slowly but surely and we still see the archives showing up in results. It's been around a week.

Is there something else we should be doing, or is it just a waiting game?


It will be a matter of waiting for the pages to be recrawled and the noindex seen.

I would not block by robots.txt as this is not a definite method of removing already indexed pages, allow the pages to be crawled for the noindex tag to be seen.

If I block Google from crawling a page using a robots.txt disallow directive, will it disappear from search results?


| improve this answer | |

You should disallow your archives in the robots.txt file.

| improve this answer | |
  • 1
    Since Google already has it in their index, won't disallowing it mean they won't see the noindex, hence it won't change their index? – psorensen Apr 7 '14 at 14:25
  • Not sure on noindex tag, but disallowing in robots.txt will exclude URLs from being indexed for sure. After a couple of days or a week URLs that have been already indexed will be excluded from Google index. – Oleg Apr 7 '14 at 14:58
  • Do not be surprised if this process takes 30-60 days depending. Google can be a bit slow and there is a lag time between spidering a page and indexing the same page. Worse yet, it can take a while for Google to drop a page, but they are very good about it. If they see a bunch of drops, they sometimes speed up the respidering of the site or reading and processing the sitemap. Oleg is right. If at all possible restrict using robots.txt first since Google seems to rely on this the most. Then noindex. Both is fine. I like shotgun approaches. – closetnoc Apr 7 '14 at 15:18
  • 1
    In every case I have seen, if a URL is already indexed in Google, when you block it via robots.txt, the URL will remain in Google, but it will say 'A description for this result is not available because of this site's robots.txt – learn more' and will most likely drop into the omitted results. I prefer meta robots 'NOINDEX' as it is cleaner and will actually drop the URL from the index. – Max Apr 7 '14 at 22:59
  • 1
    Removing urls from a sitemap is not a method to remove urls from the index. – user29671 Apr 8 '14 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.