2

We are having issues getting our new pages crawled by Google and cannot figure out why. We've done lots of research and are still having trouble finding answers. Here's some information regarding our current situation:

  • 10 million pages indexed, and tens of thousands of webpages added daily
  • updated web site
  • virtually no crawl for "new pages" / only re-crawl (or checking) of old pages - 5 million pages checked
  • We doubt google crawler has recognized our changes and are worried the Crawler might think it has to re-crawl all of the older pages as well.
  • rolled back web site last week
  • still no crawl for "new pages"

Any help on what we should do for our new pages to be crawled?

Improve this question

3 Answers 3

Reset to default
1

You could try and promote the site a bit more and get more inbound links, and/or ensure that your server can cope with a bigger crawl. See below, (but the whole answer linked is worth a read) :

There is also not a hard limit on our crawl. The best way to think about it is that the number of pages that we crawl is roughly proportional to your PageRank. So if you have a lot of incoming links on your root page, we'll definitely crawl that. Then your root page may link to other pages, and those will get PageRank and we'll crawl those as well. As you get deeper and deeper in your site, however, PageRank tends to decline.

Another way to think about it is that the low PageRank pages on your site are competing against a much larger pool of pages with the same or higher PageRank. There are a large number of pages on the web that have very little or close to zero PageRank. The pages that get linked to a lot tend to get discovered and crawled quite quickly. The lower PageRank pages are likely to be crawled not quite as often

http://www.stonetemple.com/articles/interview-matt-cutts-012510.shtml

Improve this answer
1

If your site already had 10 million pages indexed, I would say double-check that your site is crawlable and a sitemap is submitted through Google Webmaster Tools. Google does prefer sitemaps for larger sites such as yours.

It takes time for Google to notice changes sometimes especially if there is a major change with a ton of pages. It can take as much as 30 days or even more sometimes before Google figures out that a major change has happened. It will sample your site then index aggressively depending upon what it sees.

I would stay the course with your updated site making sure Google can crawl it. The more gyrations Google sees, the slower it goes sometimes. So rolling back the site may actually slow things down.

Improve this answer
4
  • Thanks for your comment and trying to help. We already submitted the sitemap to Google Webmaster Tools back in March.
    – Jeremy29
    Nov 21, 2014 at 3:04
  • @Jeremy29 I made a bit of a radical change to a site just about a year ago. It took Google time to notice that page after page seemed to change, disappear, redirect, and so on. Then Google sampled my site using the sitemap and some links that it already knew about. After a while, Google made a complete re-fetch of the entire site both from the sitemap and previous pages. Since then I have done this a few times to Google. It can take about a couple of months for Google to notice the change and think to itself- I better re-index this site.
    – closetnoc
    Nov 21, 2014 at 4:45
  • interesting.. ok so how long did it take to re-fetch all of your urls?
    – Jeremy29
    Nov 21, 2014 at 6:38
  • I had about 500,000 - 600,000 pages. It took about a month or two before Google really noticed the change, then they indexed about 48,000-53,000 pages a day until about 30,000 pages before finishing then they began to slow down progressively for the last 30k. I have done this a few times as my site went well over 1 million pages, then I dropped half, then added a few hundred thousand and so on. Each time, there was a lag of about 1-2 months then a complete re-index. I am about to do it again! Poor Google...
    – closetnoc
    Nov 21, 2014 at 14:52
0

set up a webmaster for your website, it is recommended by google. It will help you in finding errors in crawling.

Improve this answer
3
  • We have already set up Google Webmasters and can't find answers within it
    – Jeremy29
    Nov 20, 2014 at 6:39
  • check your website have any iframes, cause google unable to crawl iframes.
    – Elin Olsson
    Nov 20, 2014 at 7:09
  • We do not use any iFrames..
    – Jeremy29
    Nov 21, 2014 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.