1

I read elsewhere on here that one can't re-index their site with Google, but I like to challenge that.

Currently, I have every page on my website (that contains about 1,000,000 pages) labeled with a noarchive robots meta tag so that Google can stop caching old webpage content.

Once all the old content is no longer cached I will remove the noarchive robots tag so that Google can cache the NEW improved pages.

I'm just curious of how long this process will usually take. In webmaster tools, I set it to allow Google to max setting which is to crawl up to two pages per second.

At the moment of the 1,000,000 pages, about 175,000 are indexed and it would be great if I can get Google's caches to be updated with the new pages.

Can someone give me an idea how long the process takes?

Improve this question
3
  • It varies by site. There is no set timeframe.
    – John Conde ♦
    Commented Apr 18, 2015 at 0:17
  • From what I have seen I can give a very rough, off the hip guess: cache usually lags behind line data by 1 month to 5 days depending on the popularity, freshness, and reach of the page. If you have a million pages, the lag will vary immensely. To be safe, I would do 1 month minimum to clear them. Then again, once you want to repeat the process it can become Errrrr. Look to Cloudflare "always online" for a more thoroughly procured alternative.
    – dhaupin
    Commented Apr 18, 2015 at 0:37
  • Google can take it's sweet time sometimes, then all of a sudden, it begins to systematically crawls most of your entire site in just a few days or weeks. It seems like feast or famine sometimes. But there is a method to their madness. If your pages were not previously in your sitemap and you add them, it may help- not entirely sure. But you may not have a choice and will have to wait for that massive re-fetching of your site. I have been waiting for just the same thing. Meanwhile, Google will taste your site once it discovers a bunch of pages have changed and that may help.
    – closetnoc
    Commented Apr 18, 2015 at 0:40

1 Answer 1

Reset to default
1

Firstly there is no need to use the noarchive robots tag in your situation. You state it is to remove all of your old content from the Google cache and then you remove it to have the cache save your new content. It doesn't exactly work this way as each time Google detects a new version of your page the cached copy stored with Google will be updated to reflect the new version of your page.

As for how long this takes it very much depends. Google can take a short time or a long time to re-index your site, and more often than not Google won't re-index your whole site at once, rather they will re-index different parts of it at different rates depending on what the re-indexing algorithms deem necessary.

On top of setting the index frequency in webmaster tools to a high level, the only other solution would be to have all of the pages listed in a sitemap.xml file, then each time you update a page you update the lastmod declaration in the sitemap url record. You can also set through the sitemap how frequently the page in question is expected to be updated which Google does use in evaluating how frequently to re-index your site. Once the changes have been made to your sitemap.xml file you re-upload the sitemap.xml file to Google Webmaster Tools and Google will then process the sitemap and add the necessary pages to the crawl queue. Doesn't necessarily speed up the process per-say but it is a faster process than waiting for natural linking to trigger a re-index of your page.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.