Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We are currently developing a site that currently has 8 million unique pages that will grow to about 20 million right away, and eventually to about 50 million or more.

Before you criticize... Yes, it provides unique, useful content. We continually process raw data from public records and by doing some data scrubbing, entity rollups, and relationship mapping, we've been able to generate quality content, developing a site that's quite useful and also unique, in part due to the breadth of the data.

It's PR is 0 (new domain, no links), and we're getting spidered at a rate of about 500 pages per day, putting us at about 30,000 pages indexed thus far. At this rate, it would take over 400 years to index all of our data.

I have two questions:

  1. Is the rate of the indexing directly correlated to PR, and by that I mean is it correlated enough that by purchasing an old domain with good PR will get us to a workable indexing rate (in the neighborhood of 100,000 pages per day).
  2. Are there any SEO consultants who specialize in aiding the indexing process itself. We're otherwise doing very well with SEO, on-page especially, besides, the competition for our "long-tail" keyword phrases is pretty low, so our success hinges mostly on the number of pages indexed.

Our main competitor has achieved approx 20MM pages indexed in just over one year's time, along with an Alexa 2000-ish ranking.

Noteworthy qualities we have in place:

  • page download speed is pretty good (250-500 ms)
  • no errors (no 404 or 500 errors when getting spidered)
  • we use Google webmaster tools and login daily
  • friendly URLs in place
  • I'm afraid to submit sitemaps. Some SEO community postings suggest a new site with millions of pages and no PR is suspicious. There is a Google video of Matt Cutts speaking of a staged on-boarding of large sites, too, in order to avoid increased scrutiny (at approx 2:30 in the video).

  • Clickable site links deliver all pages, no more than four pages deep and typically no more than 250(-ish) internal links on a page.
  • Anchor text for internal links is logical and adds relevance hierarchically to the data on the detail pages.
  • We had previously set the crawl rate to the highest on webmaster tools (only about a page every two seconds, max). I recently turned it back to "let Google decide" which is what is advised.

share|improve this question
3  
I would really like to see 50 million pages providing unique useful contents. It's cool Wikipedia lacks so much knowledge compared to your site it's only 3.5 millions pages today [ref. en.wikipedia.org/wiki/File:EnwikipediaArt.PNG] –  Marco Demaio Nov 11 '10 at 14:08
1  
:) Looking past the sarcasm... it is not the number of pages which makes Wikipedia a tremendous source of knowledge -- clearly, yes -- more useful. Our site generates a page for every record of a person and a page for every record of a company in our database. We use data analysis and scrubbing to dynamically generate relationships among business partners, graphically depicting a business network of related people and corporations. The number of pages is a function of the amount of data we have. Making it discoverable via search makes it more useful to all. Thank you for your comment. –  Chris Adragna Nov 11 '10 at 14:31
    
Get more pagerank, by getting more inlinks. Link to your pages from pages that have pagerank. –  Alex Black Nov 13 '10 at 0:43

3 Answers 3

up vote 12 down vote accepted

Some potential strategies:

  • Google Webmaster Tools allows you to request an increased crawl rate. Try doing that if you haven't already.
  • Take another look at your navigation architecture to see if you can't improve access to more of your content. Look at it from a user's perspective: If it's hard for a user to find a specific piece of information, it may be hard for search engines as well.
  • Make sure you don't have duplicate content because of inconsistent URL parameters or improper use of slashes. By eliminating duplicate content, you cut down on the time Googlebot spends crawling something it has already indexed.
  • Use related content links and in-site linking within your content whenever possible.
  • Randomize some of your links. A sidebar with random internal content is a great pattern to use.
  • Use dates and other microformats.
  • Use RSS feeds wherever possible. RSS feeds will function much the same as a sitemap (in fact, Webmaster Tools allows you to submit a feed as a sitemap).
  • Regarding sitemaps, see this question.
  • Find ways to get external links to your content. This may accelerate the process of it getting indexed. If it's appropriate to the type of content, making it easy to share socially or through email will help with this.
  • Provide an API to incentivize use of your data and external links to your data. You can have an attribution link as a requirement to the data use.
  • Embrace the community. If you reach out to the right people in the right way, you'll get external links via blogs and Twitter.
  • Look for ways to create a community around your data. Find a way to make it social. API's, mashups, social widgets all help, but so do a blog, community showcases, forums, and gaming mechanics (also, see this video).
  • Prioritize which content you have indexed. With that much data, not all of it is going to be absolutely vital. Make a strategic decision as to what content is most important, e.g., it will be most popular, it has the best chance at ROI, it will be the most useful, etc. and make sure that that content is indexed first.
  • Do a detailed analysis of what your competitor is doing to get their content indexed. Look at their site architecture, their navigation, their external links, etc.

Finally, I should say this. SEO and indexing are only small parts to running a business site. Don't lose focus on ROI for the sake of SEO. Even if you have a lot of traffic from Google, it doesn't matter if you can't convert it. SEO is important, but it needs to be kept in perspective.

Edit:

As an addendum to your use case: you might consider offering reviews or testimonials for each person or business. Also, giving out user badges like StackOverflow does could entice at least some people to link to their own profile on your site. That would encourage some outside linking to your deep pages, which could mean getting indexed quicker.

share|improve this answer
1  
+1 - Always bears mentioning that SEO is a microcosm of the greater problem of promoting business; it's also the easiest (for technically-minded people, at least) to get lost in. More people are watching TV right now than at any prior point in time in history - depending upon what you're offering, TV advertising may have a better ROI than PPC... –  danlefree Nov 11 '10 at 0:57
    
Good idea on the microformats. It's not a silver bullet, but we have a number of places where semantic markup using microformat standards in place could be helpful. –  Chris Adragna Nov 11 '10 at 21:09
    
I'm don't know if any microformats (other than possibly a timestamp) would mean getting indexed quicker, but there may be other SEO benefits in using them. At the very least, it makes your pages easier to crawl and can help make your entry stand out on the search engine results page (depending on the microformat). –  Virtuosi Media Nov 11 '10 at 21:32

How to get tens of millions of pages indexed by Google bot?

It won't happen overnight, however, I guarantee that you would see more of your pages spidered sooner if inbound links to deep content (particularly sitemap pages or directory indexes which point to yet deeper content) were being added from similarly-large sites which have been around for a while.

Will an older domain be sufficient to get 100,000 pages indexed per day?

Doubtful, unless you're talking about an older domain that has had a significant amount of activity on it (i.e. accumulated content and inbound links) over the years.

Are there any SEO consultants who specialize in aiding the indexing process itself.

When you pose the question that way, I'm sure you'll find plenty of SEO's who loudly proclaim "yes!" but, at the end of the day, Virtuosi Media's suggestions are as good advice as you'll get from any of them (to say nothing of the potentially-bad advice).

From the sound of it, you should consider utilizing business development and public relations channels to build your site's ranking at this point - get more links to your content (preferably by partnering with an existing site which offers regionally-targeted content to link in to your regionally-divided content, for example), get more people browsing to your site (some will have the Google toolbar installed so their traffic may work toward page discovery), and, if possible, get your business talked about on the news or in communities of people who have a need for it (if you plan to charge for certain services, consider advertising a free trial period to draw interest).

share|improve this answer

Gaming the system is never a good idea if you're running a legitimate business that values its online reputation. Also, if your site genuinely provides value, then the longer it's around (I assume you're doing some form of marketing?) the more backlinks it will accrue, so your PR will go up and your crawl rate will go up.

Also, if you have a good link structure on your site (all of your pages are discoverable in a reasonable number of clicks/links), then you only need to submit the main indexes via sitemap. Once those pages are indexed by Google, they will be crawled by Google, and Google will index the rest of the pages on its own.

share|improve this answer
    
+1 RE: gaming the system - though I think it should be noted that there are plenty of alternatives to gaming the system which allow a webmaster to generate legitimate backlinks (which will be useful to visitors) to his or her site. –  danlefree Nov 11 '10 at 0:52
    
@danlefree: Definitely. I was only referring to buying expired domain names to get their residual PR/traffic. But if you can advertise your site, send out press releases to trade publications, application review sites, etc. then those are very good ways to generate legitimate backlinks. –  Lèse majesté Nov 11 '10 at 0:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.