I built a digg-like clone at fantasybookmark.com. Google crawled the site, but only indexed about 21 pages out of the 2,000 or so (and growing) pages on the site. We use rss feeds to find and generate content and allow users to comment.

What are reasons we are not getting indexed? Are there ways to solve this?

  • How old is the site? Have you submitted an XML sitemap? Are you reposting content other websites? If so that's duplicate content. – John Conde Oct 1 '10 at 16:17
  • RSS Feed - site is about two weeks old, and has been indexed. We take rss feeds from other content providers, yes. We also have commenting on websites. Could the issue be that we're not generating comments? At what point does duplicate content not count? - We probably don't display more than 50 words of duplicate content per entry. Is there a way we can use an algorithm to switch words so it won't be seen as duplicate content? I guess the question is - if duplicate is the issue - how different does content need to be so that it's no longer an issue. – Bob Cavezza Oct 1 '10 at 16:19

Your site appears to be quite new and full of low-quality content. For 90% of the pages, the only unique content is a title and one-line description. It may have been better to start with a small group of hand-picked articles on which to generate discussion and build a community.

A brand new site with thousands of pages can sometimes ring alarm bells with Googlebot. There have been cases of sites generating masses of unoriginal content and managing to get ranked for obscure long tail keywords even though the site is full of spam essentially.

All the usual advice will apply here: create an XML sitemap, check your site in Google Webmaster Tools. You could also try increasing your crawl rate in GWT.

| improve this answer | |
  • Thanks - I think I'll take your advice. Will probably be better to generate long tails by adding a few links every day than getting to 1000 quickly and getting zero indexing. Appreciate the help - thanks! – Bob Cavezza Oct 1 '10 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy