0

I'm looking for the best practice to let Google know about a vast number of dynamically created content. Let's say (I mean - dream) that I'm Facebook, and I want to let Google to index all the users' posts.

  1. Sitemap.xml may be the answer for this but they are limited to 50,000 URLs in each site map. I know that I can create 500 sitemaps and create a sitemap for sitemaps, but they are also limited, 25,000,000 URLS sounds quite enough at the moment, but could cause problems in the future. I.E - Stack Overflow already has 3 Million posts, probably sitemap is not the solution for them.

  2. Creating a page with paging, and links to all the dynamic data. i guess this is what Stack Overflow did by creating this page here: https://stackoverflow.com/questions

So I think that Option 2 is the answer, but it seems to me that sitemaps might have some added value. So what should I do?

Improve this question
3
  • Are there posts that aren't not going to be linked to from any other page?
    – paulmorriss
    Commented Jun 21, 2012 at 10:43
  • 1
    [this would be a comment, but I cannot comment yet] See what stackoverflow thinks: webmasters.stackexchange.com/questions/4803/the-sitemap-paradox
    – milo5b
    Commented Jun 21, 2012 at 10:52
  • paulmorriss - the posts are serverd via ajax, so google isn't able to crawl them, unless i will make some special steps to do so. so I thought that by creating a sitemap with all the links to the posts, it will make my website crawlable. I guess i was wrong, after I read milo5b excelent link: "The Sitemap file isn't meant to "fix" crawlability issues. If your site can't be crawled, fix that first".
    – Yaniv
    Commented Jun 21, 2012 at 13:50

1 Answer 1

Reset to default
1

Both options are what you should be striving for. You want to make it easy for your users and search engines to find your content. As for the XML sitemap, if your goal is to become as large as Facebook then you better be prepared to deal with things on a large scale and that includes sitemaps.

Improve this answer
1
  • my users will see my content that served to them nicely via AJAX, it's only the Search Engines that i'm concerned of. what do you mean by "you better be prepared", by what? by doing a 500 sitemaps for 50,000 urls each? maybe i should create a site map and create my AJAX website crawlable? I can think about different way rather than AJAX to do it.
    – Yaniv
    Commented Jun 21, 2012 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.