I built a digg-like clone at fantasybookmark.com. Google crawled the site, but only indexed about 21 pages out of the 2,000 or so (and growing) pages on the site. We use rss feeds to find and generate content and allow users to comment.

What are reasons we are not getting indexed? Are there ways to solve this?

  • How old is the site? Have you submitted an XML sitemap? Are you reposting content other websites? If so that's duplicate content.
    – John Conde
    Commented Oct 1, 2010 at 16:17
  • RSS Feed - site is about two weeks old, and has been indexed. We take rss feeds from other content providers, yes. We also have commenting on websites. Could the issue be that we're not generating comments? At what point does duplicate content not count? - We probably don't display more than 50 words of duplicate content per entry. Is there a way we can use an algorithm to switch words so it won't be seen as duplicate content? I guess the question is - if duplicate is the issue - how different does content need to be so that it's no longer an issue.
    – Bob Cavezza
    Commented Oct 1, 2010 at 16:19

1 Answer 1


Your site appears to be quite new and full of low-quality content. For 90% of the pages, the only unique content is a title and one-line description. It may have been better to start with a small group of hand-picked articles on which to generate discussion and build a community.

A brand new site with thousands of pages can sometimes ring alarm bells with Googlebot. There have been cases of sites generating masses of unoriginal content and managing to get ranked for obscure long tail keywords even though the site is full of spam essentially.

All the usual advice will apply here: create an XML sitemap, check your site in Google Webmaster Tools. You could also try increasing your crawl rate in GWT.

  • Thanks - I think I'll take your advice. Will probably be better to generate long tails by adding a few links every day than getting to 1000 quickly and getting zero indexing. Appreciate the help - thanks!
    – Bob Cavezza
    Commented Oct 1, 2010 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.